Hence, it behaves like an ETL if-else expression.Įxclusive choice can be useful in different situations. Instead of following all branches, it chooses a single one to follow. The last pattern also uses branches but it does the opposite of 2 previous ones. Read_input > aggregate_data > convert_to_parquet Read_input = DummyOperator(task_id='read_input_hour_'.format(hour)) The difference is that at the end one task waits for all upstream tasks to terminate: Synchronization pattern in Apache Airflow still uses multiple downstream tasks.

Thanks to it, you take advantage of parallel processing of branches and still end up with a single process conciliating the branches at the end.Īs you can see in the schema, we're generating Parquet files daily and at the end of the generation, we move all of them into an appropriate partition directory. Synchronization pattern will take the output generated in each of the branches and reconcile it. The 3rd pattern is the follow-up for parallel split. To implement parallel split in Apache Airflow you can simply set multiple downstream tasks:Ĭonvert_to_parquet = DummyOperator(task_id='convert_to_parquet')Ĭonvert_to_avro = DummyOperator(task_id='convert_to_avro') Later, this dataset is processed by 2 branches where one of them converts the data into Apache Parquet files whereas another one into Apache Avro files. Since it's a little bit more complex than the previous pattern, let's see it in an image:Īs you can see, the pipeline starts by generating some dataset. This pattern is helpful when you need to load one dataset into multiple different places, with different formats but within the same pipeline. Here the sequential character of the pipeline is replaced by a parallel execution of two or multiple branches. Read_input > aggregate_data > write_to_redshift Write_to_redshift = DummyOperator(task_id='write_to_redshift') Read_input = DummyOperator(task_id='read_input')Īggregate_data = DummyOperator(task_id='generate_data') Since there is no magic here, I will give you a simple implementation in Apache Airflow just here: The first pattern has not a lot of mystery since it represents a simple sequence of different tasks, executed one after another. In the third part, you will discover synchronization pattern whereas in the last one, exclusive choice pattern. The next section talks about a pattern called parallel split. The first presented pattern is sequential pattern, which is the simplest from the 4 patterns. Each section describes one ETL pattern illustrated with an example of an Apache Airflow DAG.

Airflow etl