Create: 2013/03/27
LastUpdate: 2013/05/20
下図の環境を使用します。
環境の詳細を知りたい場合は、メニューに戻って構築手順を参照してください。
ここでは、JobSchedulerの内部APIと同期の機能を利用して下図のジョブフローを実現してみます。
手順については、「JobScheduler FAQ」を参考にしました。
このジョブフローを、JobSchduler で、下図のように定義します。
- ParallelSample ・・・ メイン処理のJobChain。これを実行すると全てのジョブが実行される。
- ParallelExecution ・・・ executer ジョブから内部APIを使用して実行されるJobChain。2パターンの流れの state を定義
ジョブ関連の定義と格納するホットフォルダは以下のとおり
- /home/jobs/sos-berlin.com/jobscheduler/scheduler/config/live/test/event2
/home/jobs/sos-berlin.com/jobscheduler/scheduler/config/live/test/event2 ├── ParallelExecution.job_chain.xml ├── ParallelSample.job_chain.xml ├── executer.job.xml ├── job1.job.xml ├── job2.job.xml ├── job3.job.xml ├── jobA.job.xml ├── jobB.job.xml └── sync.job.xml
1.sync の定義
同期ジョブを作成します。ファイル名は以下のとおり。
- sync.job.xml
パラメータで同期に必要なOrder数を指定します。
赤字部分は、JobChainの名称です。パラメータの詳細は、JOCでこのジョブを右クリックし、メニューから[ShowDescprition]を選択してドキュメントを参照してください。
<?xml version="1.0" encoding="ISO-8859-1"?> <job order="yes" tasks="1"> <description> <include file="jobs/JobSchedulerSynchronizeJobChains.xml"/> </description> <params> <param name="ParallelExecution_required_orders" value="2"/> <param name="ParallelSample_required_orders" value="1"/> </params> <script language="java" java_class="sos.scheduler.job.JobSchedulerSynchronizeJobChains" java_class_path=""/> <run_time/> </job>
2.ParallelExecution の定義
execution ジョブ(後述)から実行される JobCahin を定義します。ファイル名は以下のとおり。
- ParallelExecution.job_chain.xml
Order生成時に "p1"または"p2"のどちらから実行するかを設定します。
<?xml version="1.0" encoding="ISO-8859-1"?> <job_chain orders_recoverable="yes"> <job_chain_node state="p1" job="job1" next_state="p1.b" error_state="error"/> <job_chain_node state="p1.b" job="job2" next_state="sync" error_state="error"/> <job_chain_node state="p2" job="job3" next_state="sync" error_state="error"/> <job_chain_node state="sync" job="sync" next_state="success" error_state="error"/> <job_chain_node state="success"/> <job_chain_node state="error"/> </job_chain>job1、job2、job3 は以下のように定義します。
テスト時にJOCの画面を確認しやすいように10秒スリープします。
<?xml version="1.0" encoding="ISO-8859-1"?> <job order="yes" stop_on_error="no"> <script language="shell"> <![CDATA[ echo "$SCHEDULER_JOB_NAME processed $SCHEDULER_ORDER_ID" echo "Current Timestamp: `date '+%Y-%m-%d %H:%M:%S'`" echo "Current Hostname: `uname -n`" echo "Current Username: `whoami`" sleep 10 exit 0 ]]> </script> <run_time/> </job>
3.ParallelSample の定義
メインのJobChainを定義します。ファイル名は以下のとおり。
- ParallelSample.job_chain.xml
executer の state (赤字)は ParallelExecute の開始 state として使用されます。
<?xml version="1.0" encoding="ISO-8859-1"?> <job_chain orders_recoverable="yes" visible="yes"> <job_chain_node state="100" job="jobA" next_state="p1" error_state="error"/> <job_chain_node state="p1" job="executer" next_state="p2" error_state="error"/> <job_chain_node state="p2" job="executer" next_state="sync" error_state="sync"/> <job_chain_node state="sync" job="sync" next_state="200" error_state="error"/> <job_chain_node state="200" job="jobB" next_state="success" error_state="error"/> <job_chain_node state="success"/> <job_chain_node state="error"/> </job_chain>executer ジョブは以下のとおり。
パラメータで指定された JobChain から Order を生成します。
また、job_chain_node の state に設定された値(上記の赤字)を、Order の開始 state として設定します。
なお、このジョブは、Order を生成&実行するだけで、Orderの実行が完了するまで待機することはありません。
<?xml version="1.0" encoding="ISO-8859-1"?> <job order="yes" stop_on_error="no" tasks="3"> <params> <param name="exe_jobchain" value="/test/event2/ParallelExecution"/> </params> <script language="javascript"> <![CDATA[ function spooler_process(){ var thisOrder = spooler_task.order; var order = spooler.create_order(); var exe_jobchain = spooler_task.params.value("exe_jobchain"); spooler_log.info("** exe_jobchain:" + exe_jobchain); order.state = thisOrder.state; spooler_log.info("** exe_state:" + order.state); order.params.merge(thisOrder.params); order.id = thisOrder.id + "_" + thisOrder.state; spooler_log.info("** Order addes:" + order.id); spooler.job_chain( exe_jobchain ).add_order( order ); return true; } ]]> </script> <monitor name="configuration_monitor" ordering="0"> <script java_class="sos.scheduler.managed.configuration.ConfigurationOrderMonitor" language="java" java_class_path=""/> </monitor> <run_time/> </job>jobA、jobBの内容は以下のとおり。
<?xml version="1.0" encoding="ISO-8859-1"?> <job order="yes" stop_on_error="no"> <script language="shell"> <![CDATA[ echo "$SCHEDULER_JOB_NAME processed $SCHEDULER_ORDER_ID" echo "Current Timestamp: `date '+%Y-%m-%d %H:%M:%S'`" echo "Current Hostname: `uname -n`" echo "Current Username: `whoami`" exit 0 ]]> </script> <run_time/> </job>
4.動作確認
JOC で ParallelSample を実行します。
今回は、上で紹介した図に合わせるために、Order ID に "123" を設定します。
下図は、実行中の画面です。
ParallelSample は、jobBの直前で中断しています。
PrallelExecution は、"p1"から開始した Order と、"p2" から開始した Order が実行中です。
PrallelExecution の2つのOrder が sync ジョブまで実行したら、jobB が実行されて正常終了します。
下図は、正常終了後に、実行時間を確認したところです。
PrallelExecution のOrder ID である "123_p1" と "123_p2" は、開始時間が同じなのでパラレルに実行していたことがわかります。
ParallelSample の実行結果は以下のとおり。
2013-03-27 16:43:45.633 [info] (Task test/event2/jobA:4921) SCHEDULER-842 Task is going to process Order test/event2/ParallelSample:123, state=100, on Scheduler http://centos6:4444 2013-03-27 16:43:45.634 [info] (Task test/event2/jobA:4921) 2013-03-27 16:43:45.634 [info] (Task test/event2/jobA:4921) Task test/event2/jobA:4921 - Protocol starts in /home/jobs/sos-berlin.com/jobscheduler/scheduler/logs/task.test,event2,jobA.log 2013-03-27 16:43:45.635 [info] (Task test/event2/jobA:4921) SCHEDULER-918 state=starting (at=never) 2013-03-27 16:43:45.636 [info] (Task test/event2/jobA:4921) SCHEDULER-987 Starting process: '/bin/sh' '-c' '"/tmp/jobs/sos.lPLnc3"' 2013-03-27 16:43:45.725 [info] (Task test/event2/jobA:4921) jobA processed 123 2013-03-27 16:43:45.725 [info] (Task test/event2/jobA:4921) Current Timestamp: 2013-03-27 16:43:45 2013-03-27 16:43:45.725 [info] (Task test/event2/jobA:4921) Current Hostname: centos6 2013-03-27 16:43:45.725 [info] (Task test/event2/jobA:4921) Current Username: jobs 2013-03-27 16:43:45.726 [info] (Task test/event2/jobA:4921) SCHEDULER-915 Process event 2013-03-27 16:43:45.727 [info] (Task test/event2/jobA:4921) SCHEDULER-843 Task has ended processing of Order test/event2/ParallelSample:123, state=100, on Scheduler http://centos6:4444 2013-03-27 16:43:45.727 [info] set_state p1, Job /test/event2/executer 2013-03-27 16:43:45.764 [info] (Task test/event2/executer:4922) SCHEDULER-842 Task is going to process Order test/event2/ParallelSample:123, state=p1, on Scheduler http://centos6:4444 2013-03-27 16:43:45.765 [info] (Task test/event2/executer:4922) 2013-03-27 16:43:45.765 [info] (Task test/event2/executer:4922) Task test/event2/executer:4922 - Protocol starts in /home/jobs/sos-berlin.com/jobscheduler/scheduler/logs/task.test,event2,executer.4922.log 2013-03-27 16:43:45.794 [info] (Task test/event2/executer:4922) SCHEDULER-918 state=starting (at=never) 2013-03-27 16:43:46.220 [info] (Task test/event2/executer:4922) Configuration File: /home/jobs/sos-berlin.com/jobscheduler/scheduler/config/live/test/event2/ParallelSample.config.xml not found (Probably running on an agent). 2013-03-27 16:43:46.220 [info] (Task test/event2/executer:4922) Reading configuration from xml payload... 2013-03-27 16:43:46.233 [WARN] (Task test/event2/executer:4922) ConfigurationBaseMonitor: Monitor: error occurred preparing configuration: Premature end of file. 2013-03-27 16:43:46.238 [info] (Task test/event2/executer:4922) ** exe_jobchain:/test/event2/ParallelExecution 2013-03-27 16:43:46.239 [info] (Task test/event2/executer:4922) ** exe_state:p1 2013-03-27 16:43:46.240 [info] (Task test/event2/executer:4922) ** Order addes:123_p1 2013-03-27 16:43:46.282 [info] (Task test/event2/executer:4922) SCHEDULER-843 Task has ended processing of Order test/event2/ParallelSample:123, state=p1, on Scheduler http://centos6:4444 2013-03-27 16:43:46.283 [info] set_state p2, Job /test/event2/executer 2013-03-27 16:43:46.297 [info] (Task test/event2/executer:4924) SCHEDULER-842 Task is going to process Order test/event2/ParallelSample:123, state=p2, on Scheduler http://centos6:4444 2013-03-27 16:43:46.297 [info] (Task test/event2/executer:4924) 2013-03-27 16:43:46.297 [info] (Task test/event2/executer:4924) Task test/event2/executer:4924 - Protocol starts in /home/jobs/sos-berlin.com/jobscheduler/scheduler/logs/task.test,event2,executer.4924.log 2013-03-27 16:43:46.332 [info] (Task test/event2/executer:4924) SCHEDULER-918 state=starting (at=never) 2013-03-27 16:43:46.740 [info] (Task test/event2/executer:4924) Configuration File: /home/jobs/sos-berlin.com/jobscheduler/scheduler/config/live/test/event2/ParallelSample.config.xml not found (Probably running on an agent). 2013-03-27 16:43:46.740 [info] (Task test/event2/executer:4924) Reading configuration from xml payload... 2013-03-27 16:43:46.754 [WARN] (Task test/event2/executer:4924) ConfigurationBaseMonitor: Monitor: error occurred preparing configuration: Premature end of file. 2013-03-27 16:43:46.759 [info] (Task test/event2/executer:4924) ** exe_jobchain:/test/event2/ParallelExecution 2013-03-27 16:43:46.760 [info] (Task test/event2/executer:4924) ** exe_state:p2 2013-03-27 16:43:46.761 [info] (Task test/event2/executer:4924) ** Order addes:123_p2 2013-03-27 16:43:46.803 [info] (Task test/event2/executer:4924) SCHEDULER-843 Task has ended processing of Order test/event2/ParallelSample:123, state=p2, on Scheduler http://centos6:4444 2013-03-27 16:43:46.803 [info] set_state sync, Job /test/event2/sync 2013-03-27 16:43:46.818 [info] (Task test/event2/sync:4926) SCHEDULER-842 Task is going to process Order test/event2/ParallelSample:123, state=sync, on Scheduler http://centos6:4444 2013-03-27 16:43:46.819 [info] (Task test/event2/sync:4926) 2013-03-27 16:43:46.819 [info] (Task test/event2/sync:4926) Task test/event2/sync:4926 - Protocol starts in /home/jobs/sos-berlin.com/jobscheduler/scheduler/logs/task.test,event2,sync.log 2013-03-27 16:43:46.853 [info] (Task test/event2/sync:4926) SCHEDULER-918 state=starting (at=never) 2013-03-27 16:43:47.348 [info] (Task test/event2/sync:4926) synchronizing order [123, ] of job chain [/test/event2/ParallelSample] 2013-03-27 16:43:47.362 [info] (Task test/event2/sync:4926) job chain [/test/event2/ParallelExecution], state [sync] requires 2 orders being present, 0 orders have been enqueued 2013-03-27 16:43:47.366 [info] (Task test/event2/sync:4926) job chain [/test/event2/ParallelSample], state [sync] requires 1 orders being present, 1 orders have been enqueued 2013-03-27 16:43:47.367 [info] (Task test/event2/sync:4926) Synchits:0 2013-03-27 16:43:47.368 [info] SCHEDULER-991 Order has been suspended 2013-03-27 16:43:47.369 [info] (Task test/event2/sync:4926) order is set on hold: 123, 2013-03-27 16:43:47.370 [info] (Task test/event2/sync:4926) synchronized job chain: /test/event2/ParallelSample, state [sync], required orders: 1, pending orders: 1 2013-03-27 16:43:47.370 [info] (Task test/event2/sync:4926) synchronized job chain: /test/event2/ParallelExecution, state [sync], required orders: 2, pending orders: 0 2013-03-27 16:43:47.371 [info] (Task test/event2/sync:4926) SCHEDULER-843 Task has ended processing of Order test/event2/ParallelSample:123, state=sync, on Scheduler http://centos6:4444 2013-03-27 16:44:07.127 [info] set_state 200, Job /test/event2/jobB, suspended 2013-03-27 16:44:07.127 [info] SCHEDULER-992 Order ist not longer suspended, next start now 2013-03-27 16:44:07.148 [info] (Task test/event2/jobB:4930) SCHEDULER-842 Task is going to process Order test/event2/ParallelSample:123, state=200, on Scheduler http://centos6:4444 2013-03-27 16:44:07.148 [info] (Task test/event2/jobB:4930) 2013-03-27 16:44:07.148 [info] (Task test/event2/jobB:4930) Task test/event2/jobB:4930 - Protocol starts in /home/jobs/sos-berlin.com/jobscheduler/scheduler/logs/task.test,event2,jobB.log 2013-03-27 16:44:07.149 [info] (Task test/event2/jobB:4930) SCHEDULER-918 state=starting (at=never) 2013-03-27 16:44:07.149 [info] (Task test/event2/jobB:4930) SCHEDULER-987 Starting process: '/bin/sh' '-c' '"/tmp/jobs/sos.UaCdoL"' 2013-03-27 16:44:07.223 [info] (Task test/event2/jobB:4930) jobB processed 123 2013-03-27 16:44:07.223 [info] (Task test/event2/jobB:4930) Current Timestamp: 2013-03-27 16:44:07 2013-03-27 16:44:07.223 [info] (Task test/event2/jobB:4930) Current Hostname: centos6 2013-03-27 16:44:07.223 [info] (Task test/event2/jobB:4930) Current Username: jobs 2013-03-27 16:44:07.223 [info] (Task test/event2/jobB:4930) SCHEDULER-915 Process event 2013-03-27 16:44:07.224 [info] (Task test/event2/jobB:4930) SCHEDULER-843 Task has ended processing of Order test/event2/ParallelSample:123, state=200, on Scheduler http://centos6:4444 2013-03-27 16:44:07.224 [info] set_state success 2013-03-27 16:44:07.224 [info] SCHEDULER-945 No further job in job chain - order has been carried out 2013-03-27 16:44:07.224 [info] SCHEDULER-940 Removing order from job chain