V2EX = way to explore
V2EX 是一个关于分享和探索的地方
Sign Up Now
For Existing Member  Sign In
scalaer
V2EX  ›  问与答

有 akka 大佬吗, 请教一个问题

  •  
  •   scalaer · Nov 15, 2019 · 1982 views
    This topic created in 2355 days ago, the information mentioned may be changed or developed.

    用 slick 流式读取 mysql 一个大表 700w, 在 map 的时候计数,发现到 360w 就停了, 请问下这是什么原因?

    val source: Source[Event, NotUsed] = Slick.source(query1)
    
    val sink: Sink[ProducerRecord[String, Event], Future[Done]] = Producer.plainSink(p)
    
    var counter = 0;
    
    val done: Future[Done] =
            source
              .map(
                e =>
                  {
                    counter += 1
                    println(counter)
                    new ProducerRecord(topic, e.sid, EventFactory.build_event(e))
                  }
    ).runWith(sink)
    
    4 replies    2019-11-16 22:26:33 +08:00
    woodfish
        1
    woodfish  
       Nov 15, 2019
    是不是 mysql keepalive 超时了
    scalaer
        2
    scalaer  
    OP
       Nov 15, 2019
    @woodfish 现在定位到是 kafka producer 超时了, 这个有啥参数可以设置的吗?
    scalaer
        4
    scalaer  
    OP
       Nov 16, 2019
    @woodfish close-timeout 设置了很大, 不起作用..
    About   ·   Help   ·   Advertise   ·   Blog   ·   API   ·   FAQ   ·   Solana   ·   1077 Online   Highest 6679   ·     Select Language
    创意工作者们的社区
    World is powered by solitude
    VERSION: 3.9.8.5 · 29ms · UTC 18:37 · PVG 02:37 · LAX 11:37 · JFK 14:37
    ♥ Do have faith in what you're doing.