How to schedule pipeline using APP Engine in Google cloud scheduler

1
votes

I read google cloud scheduler to schedule my batch pipeline. https://cloud.google.com/scheduler/docs/quickstart

I want to run batch pipeline once in a day. I read that "APP ENGINE" can schedule pipeline. I tried to find out in stack-overflow and google, however I did not find how exactly app engine can schedule batch pipeline? Is there any other way for scheduling batch pipeline using cloud scheduler? One way i found out that keep pipeline running and every hour drop pub-sub message using cloud scheduler and run the job. But this is expensive and inconvenient operation.

I want to now how exactly app engine can help to schedule batch pipeline. Thanks in advance

2
google-cloud-platformcloudscheduler
Hey, you asked this question a few days ago at this link. stackoverflow.com/questions/58228055/… Its a pretty broad question. I'd recommend you think about how to make it more specific -- what have you tried, what would be a good result, any code you have working already, etc. See more at how to ask, likewise you might wan to see about what to do when no one answers – robsiemb

2 Answers

0
votes

I want to now how exactly app engine can help to schedule batch pipeline.

According to FAQ:

You can automate pipeline execution by:

  • Using Google App Engine (Flexible Environment only) or Cloud Functions.

More details on how to schedule batching pipelines using App Engine can be found here and here

Let me know if this is helpful for you.

0
votes
Still I am not clear about it. I have created app engine in the project. Added servlet in the app engine . Then created servelet                        

@WebServlet(name = "dataflowscheduler", value = "/dataflow/schedule")
public class DataflowSchedulingServlet extends HttpServlet {
  @Override
  public void doGet(HttpServletRequest req, HttpServletResponse resp) throws IOException {
    Batchpipleine.run();
  }
}    

Batchpipeline is the java class which has run method. Run method has all steps for pipeline execution like pubs reading message, windowing, storing to db etc.

Am I on right path ?

Cloud scheduler app :
Name : Test pipeline 
Description: This will trigger batch job
Frequency : */5**** 
Traget : /dataflow/schedule

Is target correct?

Is this way to run pipeline using app engine?
Do you have any guidance or process which include process to run pipeline using app engine other than  document mentioned above