Bring fork up to date with main repo. - #1
Open
TestingCodeReview wants to merge 6658 commits into
Open
Conversation
DeepCode's analysis on #8505d2 found:
Top issues
👉 View analysis in DeepCode’s Dashboard | Configure the bot |
Currently it just shows "TimeoutError: There are still unapplied migrations after 60 seconds. " which is not that helpful.
This commit adds more info to show the difference in migrations in DB and source code running `check_migrations`:
Now:
```
TimeoutError: There are still unapplied migrations after 60 seconds. Migration Head(s) in DB: {'e165e7455d70'} | Migration Head(s) in Source Code: {'a13f7613ad25'}
```
closes #15650
This masks sensitive values in logs for Connections and Variables. It behaves as follows: - Connection passwords are always masked, where-ever they appear. This means, if a connection has a password of `a`, then _every_ `a` in log messages would get replaced with `***` - "Sensitive" keys from extra_dejson are also masked. Sensitive is defined by the "existing" mechanism that the UI used, based upon the name of the key. - "Sensitive" Variables are also masked.
We are allowign doc-only-changes when releasing providers, therefore we might want to regenerate documentation for latest version of the provider packages when there are doc-only changes. The new --override-versioned flag enables that.
…nks` (#15673) Without this change it is impossible for one of the providers to depend upon the "dev"/current version of Airflow -- pip instead would try and go out to PyPI to find the version (which almost certainly wont exist, as it hasn't been released yet)
This PR fixes a case where a task would not call the on_failure_callback when there's a case of OOM. The issue was that task pid was being set at the wrong place and the local task job heartbeat was not checking the correct pid of the process runner and task. Now, instead of setting the task pid in check_and_change_state_before_execution, it's now set correctly at the _run_raw_task method
Simply more test coverage on pgbouncer in the helm chart.
…15669) If AWS's Identity-based IAM policies are in use on the cluster they token file will be mounted in to the pod (via the service account) and, prior to this change, will be owned by root. Specifying `fsGroup` makes the file group-readable by the `airflow` user. We already specify this in our helm chart, so this change is just for anyone looking at the docs.
This will be shown in logs on initialization, and flashed in UI on later scheduled refreshes. closes #15248
Docs: https://jinja.palletsprojects.com/en/2.11.x/nativetypes/ ```python >>> from jinja2 import nativetypes >>> ne = nativetypes.NativeEnvironment() >>> import pendulum >>> ne.from_string('{{ x }}').render(x=pendulum.now()) <Pendulum [2021-03-04T15:33:17.073343+00:00]> >>> ne.from_string('{{ x }}').render(x=pendulum.now().isoformat()) '2021-03-04T15:33:29.516540+00:00' >>> ne.from_string('{{ x }}').render(x="2012-10-10") '2012-10-10' ``` Current: ```python >>> environment.Environment().from_string('{{ ["w","x"] }}').render() "['w', 'x']" ``` Proposed: ```python >>> nativetypes.NativeEnvironment().from_string('{{ ["w","x"] }}').render() ['w', 'x'] ``` * Add a flag for rendering to native python objects The problem with replacing `NativeEnvironment` as `Environment` only is that `NativeEnvironment` does not raise an error on Undefined templates: https://github.com/pallets/jinja/blob/2.11.3/src/jinja2/nativetypes.py#L70-L94
) This commit adds the following things: - Add "airflowVersion" flag that will allow use to add some components that are just available or work with certain Airflow version. Example: pod_template_file is available for Airflow >= 1.10.12 - Update logic for selecting pre/post Airflow 2.0 CLI commands based on that flag - Updates stragtegy of Airflow Webserver based on that flag as the webserver in Airflow >= 2 does not need access to DAG files, hence we don't need to recreate but can have a "true" rollingUpdate - Allow overriding webserver udpate strategy
* Allow helm chart tests to run in parallel The helm chart tests are pretty slow when run sequentially. Modifying them so they can be run in parallel saves a lot of time, from 10 minutes to 3 minutes on my machine with 8 cores. The only test that needed modification was `test_pod_template_file.py`, as it temporarily moves a file into the templates directory which was causing other tests to fail as they weren't expecting any objects from that temporary file. This is resolved by giving the pod_template_file test an isolated chart directory it can modify. `helm dep update` also doesn't work when it is called in parallel, so the fixture responsible for running it now ensures we only run it one at a time. * Enable parallelism for helm unit tests in CI Co-authored-by: Kamil Breguła <mik-laj@users.noreply.github.com> Co-authored-by: Kamil Breguła <mik-laj@users.noreply.github.com>
Latest pygments version does not render properly the
syntax where `{}` is used (terraform expects this to be
separated by EOL.
Fixing typo in bash script
* Move out www connection tests * Move out www variable tests * Move out www plugin tests * Move out www pool tests * Move out mount point tests * Move out www configuration tests * Move out www redoc tests Also merge some side-effect-less tests into one file so we don't get too many tiny files. * Move out task instance list tests * Make sure base view init is in app context * Move out www helper function tests * Move out www dagrun tests * Move out www extra link tests * Move out www trigger DAG tests * Move out www rendered field tests * Always load example DAGs before running view tests * Properly handle test sessions so new tests work * Remove 'checker' in favor of plain functions
Gets rid of Airflow 1.10 in Breeze and script/configuration. We were still using occasionally the master version of Breeze to run 1.10 version of Airflow, but this madness should end now when we are approaching 2.1 release. All changes in Breeze were so far ported to 1.10 but this is about the time to finish it.
This adds a new view for displaying dependencies between DAGs. It's based on the DAG Dependencies plugin (https://github.com/ms32035/airflow-dag-dependencies) and has been updated to work with Airflow 2.0. Since quite a bit of code is common with DAG graph view, that has been externalized to a new module. Unlike the external plugin this is uses the DAG serializater to store dependencies at parse time, meaning it doesn't need to load all DAGs. Co-authored-by: Brent Bovenzi <brent.bovenzi@gmail.com> Co-authored-by: Ash Berlin-Taylor <ash_github@firemirror.com>
* Auto-apply apply_default decorator * Automatically patch warnings stacklevel inside Operator constructors This makes two changes to make the apply_defaults transparent to warnings: 1. It removes `__call__` from the Metaclass (which added one frame) 2. It "patches" warnings.warn _in the function scope_ to automatically adjust the stacklevel The reason for this is that now that apply_defaults is called "magically" from the Metaclass we're adding extra frames to the callstack. By locally patching warnings like this it means that user code doesn't have to be aware of what we've done. (I wish Python had a native way of handling this case as it crops up all the time. Perhaps I should submit a PEP for it with a better design) * Don't test provider packges against 2.0 anymore Since we have changed the dep to need Airflow 2.1, testing against 2.0 no longer makes any sense. And since 2.1 is not yet out, we have disabled this specific step for now Co-authored-by: Kamil Breguła <kamil.bregula@polidea.com>
IIntroduce a DAG Calendar View to provide visibility over the full state of the dag by displaying the aggregated dag runs' states in a calendar. This makes it possible to monitor the state of thousands of dag runs in a single view that is concise and easy to understand. It is particularly useful to monitor the state of large backfills: Each day is displayed with a color according to the dag runs' states for that day: - If at least one dag run has failed for a day, the day will be displayed as "failed". - If all dag runs have succeeded, the day will be shown as "succeeded". - If there are still running dag runs (and no failed dag run) for that day, the day will be shown as "running". A tooltip, for each day, gives the exact number of dag runs in each state for the day. Clicking on a day redirects to the tree view for that day to show the task instances for that day. Co-authored-by: Benoit Hanotte <benoit@wayve.ai> Co-authored-by: Ash Berlin-Taylor <ash_github@firemirror.com> Co-authored-by: Brent Bovenzi <brent.bovenzi@gmail.com>
We've started to receive deprecation warnings for Node 10 and this PR attempts to upgrade to recommended Node 14. Fixes: #15713
As `config`'s documentation states values are passed through `tpl`, one would expect `config.webserver.base_url` to also support templating.
Sphinx RTD theme 0.5.2. introduced limitation to docutils to account for some docutils markup change: readthedocs/sphinx_rtd_theme#1112
This change removes unnecessary dependencies from setup_requires: * docutils are not needed in setup requires and actually having them here caused harmful upgrade even if docutils are limited to <0.17 elsewhere * setup_tools should not be needed in setup_requires (by the time setup_requires are parsed, they should be already installed * bowler is not needed any more in setup.py (we got rid of it when we got rid of backport packages. Also docutils<0.17 limitation is moved to install_requires because docutils was already a transitive dependency of airflow without extras and it could be upgraded even if there is a limitation in extra.
We have now separate `apache/airflow-ci` DockerHub repo and we move all our images needed only during CI there. The images from the main `apache/airflow` remaining are: * airflow tagged and latest tagged production images * images neded by the Helm Chart
Some images may not want to use the same command/args as the community image, so expose them as parameters.
Added Kayzen to the list of companies using Apache Airflow
…n multiprocessing (#15989) Fixes #15938 multiprocessing.Pool is known to often become stuck. It causes celery_executor to hang randomly. This happens at least on Debian, Ubuntu using Python 3.8.7 and Python 3.8.10. The issue is reproducible by running test_send_tasks_to_celery_hang in this PR several times (with db backend set to something other than sqlite because sqlite disables some parallelization) The issue goes away once switched to concurrent.futures.ProcessPoolExecutor. In python 3.6 and earlier, ProcessPoolExecutor has no initializer argument. Fortunately, it's not needed because reset_signal is no longer needed because the signal handler now checks if the current process is the parent.
closes: #14260 related: #9824 When clearing task across dags using ExternalTaskMarker the dag state of the external DagRun is not set to active. So cleared tasks in the external dag will not automatically start if the DagRun is a Failed or Succeeded state. #9824 tried to fix a similar issue for subdag. But it did not fix ExternalTaskMarker. This PR fixes both. Two changes are made to fix the issue: Make clear_task_instances set DagRuns' state to dag_run_state for all the affected DagRuns. The filter for DagRun in clear_task_instances is fixed too. Previously, it made an assumption that execution_dates for all the dag_ids are the same, which is not always correct. test_external_task_marker_clear_activate is added to make sure the fix does the right thing.
…12058) ## Expected behaviour For a sensor like this, the intention of the DAG author is usually to fail the sensor if it's still not done after ten minutes. However, if the sensor fails prematurely due to other unexpected reasons (such as network outage), retry at most twice. ```python sensor = PythonSensor( task_id='sensor', python_callable=python_callable, timeout=60 * 10, retries=2, mode="reschedule", ) ``` ## Actual behaviour The actual current behaviour of Airflow is to retry when the sensor times out. So the effective timeout of the sensor becomes 60 * 10 * (retries + 1) = 30min. This often causes confusion. It also makes it impossible to achieve the expected behaviour no matter how the author configures the sensor. ## Fix This PR fixes this issue. `AirflowSensorTimeout` is now treated as immediate failure. This achieves the expected behaviour. The sensor will fail if timeout is reached. If someone really wants the previous behaviour, he can always increase the timeout. I.e instead of failing and retrying every ten minutes three times, just set the timeout to 30min.
This check was failing on master, example: https://github.com/apache/airflow/runs/2702121768#step:10:160
The test_send_tasks_to_celery_hang hangs on self-hosted runners more often than not. It's been introduced in #15989 and while the test does not usually hang on regular GitHub runners, or in case of running it locally (I could not make it fail), it does hang almost always when run on self-hosted runners. Marking it as quarantined for now. Issue #16168 created to keep track of it.
) See #16172 For now we limit the cattrs to < 1.7.0
Fixes #15005 GCSToLocalFilesystemOperator unnecessarily downloads objects when it checks object size. Co-authored-by: Pavel Kachalov <pavel_kachalov@epam.com>
Similar to https://airflow.apache.org/docs/apache-airflow-providers-google/stable/index.html we should have separate section on Guides
* Fix: GCS To BigQuery source_object #16008 Fix GCS To BigQuery source_object to accept both str and list * convert source_objects to list if not list converting source_objects to list instead of modifying the logic part * add tests
I think this was added as a DEBUG step which was forgotten to remove in #15944
…16159) Seems that MSSQL is not able to use data volume when it is mounted from tmpfs filesystem. See microsoft/mssql-docker#13 In such case, instead of mounting docker-created volume we mount a volume mounted from home directory of the user which is unlikely to be a tmpfs volume.
This is follow up of #11542 to all the missed places in the codebase.
The docs were incorrect. Bug introduced in #15183 unfortunately.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
19 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Bring fork up to date with main repo.