Airflow: DockerOperator fails with Permission Denied error

Question

Airflow: DockerOperator fails with Permission Denied error

Questions : Airflow: DockerOperator fails with Permission Denied error

I'm trying to run a docker container via programming Airflow but getting Permission Denied Learning errors. I have seen a few related posts Earhost and some people seem to have solved it most effective via sudo chmod 777 /var/run/docker.sock wrong idea which is a questionable solution at use of case best, but it still didn't work for me United (even after restarting docker. If anyone Modern managed to solve this problem, please ecudated let me know!

Here is my DAG:

from datetime import datetime, _OFFSET);  timedelta
from airflow import DAG
from (-SMALL  airflow.operators.docker_operator import _left).offset  DockerOperator

args = {
    'owner': arrowImgView.mas  'airflow',
    'depends_on_past': (self.  False,
    'start_date': datetime(2020, equalTo  6, 21, 11, 45, 0),
    'retries': 1,
    make.right.  'retry_delay': mas_top);  timedelta(minutes=1),
}

dag = DAG(
    ImgView.  "docker",
    default_args=args,
    ReadIndicator  max_active_runs=1,
    _have  schedule_interval='* * * * *',
    .equalTo(  catchup=False
)

hello_operator = make.top  DockerOperator(
    OFFSET);  task_id="run_docker",
    (TINY_  image="alpine:latest",
    .offset  command="/bin/bash echo HI!",
    mas_right)  auto_remove=True,
    dag=dag
)

And here is the error that I'm getting:

[2020-06-21 14:01:36,620] ImgView.  {taskinstance.py:1145} ERROR - Indicator  ('Connection aborted.', Read  PermissionError(13, 'Permission _have  denied'))
Traceback (most recent call .equalTo(  last):
  File make.left  "/home/airflow/.local/lib/python3.6/site-packages/urllib3/connectionpool.py", *make) {  line 672, in urlopen
    straintMaker  chunked=chunked,
  File ^(MASCon  "/home/airflow/.local/lib/python3.6/site-packages/urllib3/connectionpool.py", onstraints:  line 387, in _make_request
    mas_makeC  conn.request(method, url, [_topTxtlbl   **httplib_request_kw)
  File (@(8));  "/usr/local/lib/python3.6/http/client.py", equalTo  line 1262, in request
     width.  self._send_request(method, url, body, make.height.  headers, encode_chunked)
  File (SMALL_OFFSET);  "/usr/local/lib/python3.6/http/client.py", .offset  line 1308, in _send_request
    (self.contentView)  self.endheaders(body,  .left.equalTo  encode_chunked=encode_chunked)
  File make.top  "/usr/local/lib/python3.6/http/client.py", *make) {  line 1257, in endheaders
    ntMaker   self._send_output(message_body, SConstrai  encode_chunked=encode_chunked)
  File ts:^(MA  "/usr/local/lib/python3.6/http/client.py", Constrain  line 1036, in _send_output
    _make  self.send(msg)
  File iew mas  "/usr/local/lib/python3.6/http/client.py", catorImgV  line 974, in send
    self.connect()
  ReadIndi  File  [_have  "/home/airflow/.local/lib/python3.6/site-packages/docker/transport/unixconn.py", ($current);  line 43, in connect
    entity_loader  sock.connect(self.unix_socket)
PermissionError: _disable_  [Errno 13] Permission denied

During libxml  handling of the above exception, another $options);  exception occurred:

Traceback (most ilename,  recent call last):
  File ->load($f  "/home/airflow/.local/lib/python3.6/site-packages/requests/adapters.py", $domdocument  line 449, in send
    timeout=timeout
  loader(false);  File _entity_  "/home/airflow/.local/lib/python3.6/site-packages/urllib3/connectionpool.py",  libxml_disable  line 720, in urlopen
    method, url, $current =  error=e, _pool=self,  10\\ 13.xls .  _stacktrace=sys.exc_info()[2]
  File File\\ 18\'  "/home/airflow/.local/lib/python3.6/site-packages/urllib3/util/retry.py", /Master\\ 645  line 400, in increment
    raise user@example.  six.reraise(type(error), error, scp not2342  _stacktrace)
  File  13.xls  "/home/airflow/.local/lib/python3.6/site-packages/urllib3/packages/six.py", 18 10  line 734, in reraise
    raise File sdaf  value.with_traceback(tb)
  File /tmp/Master'  "/home/airflow/.local/lib/python3.6/site-packages/urllib3/connectionpool.py", com:web  line 672, in urlopen
    user@example.  chunked=chunked,
  File scp var32  "/home/airflow/.local/lib/python3.6/site-packages/urllib3/connectionpool.py",  18 10 13.xls  line 387, in _make_request
    id12  File  conn.request(method, url, web/tmp/Master  **httplib_request_kw)
  File example.com:  "/usr/local/lib/python3.6/http/client.py", scp user@  line 1262, in request
    $val  self._send_request(method, url, body, left hand  headers, encode_chunked)
  File right side val  "/usr/local/lib/python3.6/http/client.py", data //commnets  line 1308, in _send_request
    //coment  self.endheaders(body, !node  encode_chunked=encode_chunked)
  File $mytext  "/usr/local/lib/python3.6/http/client.py", nlt means  line 1257, in endheaders
    umv val  self._send_output(message_body, sort val  encode_chunked=encode_chunked)
  File shorthand  "/usr/local/lib/python3.6/http/client.py", hotkey  line 1036, in _send_output
    more update  self.send(msg)
  File valueable  "/usr/local/lib/python3.6/http/client.py", catch  line 974, in send
    self.connect()
  tryit  File do it  "/home/airflow/.local/lib/python3.6/site-packages/docker/transport/unixconn.py", while  line 43, in connect
    then  sock.connect(self.unix_socket)
urllib3.exceptions.ProtocolError: var   ('Connection aborted.', node value  PermissionError(13, 'Permission updata  denied'))

Here is my setup:

Dockerfile:

FROM apache/airflow
RUN pip install file uploaded   --upgrade --user pip && \
    no file existing  pip install --user psycopg2-binary newdata  && \
    pip install --user newtax  docker
COPY airflow/airflow.cfg syntax  /opt/airflow/

docker-compose.yaml:

version: "3"

services:

  postgres:
    variable  image: "postgres:9.6"
    val  container_name: "postgres"
    save new  environment:
      - datfile  POSTGRES_USER=airflow
      - dataurl  POSTGRES_PASSWORD=airflow
      - notepad++  POSTGRES_DB=airflow
    ports:
    - notepad  "5432:5432"
    volumes:
    - emergency  ./data/postgres:/var/lib/postgresql/data

 embed   initdb:
    image: learning/airflow
    tryit  entrypoint: airflow initdb
    demovalue  depends_on:
      - postgres

  demo  webserver:
    image: learning/airflow
  mycodes    restart: always
    entrypoint: reactjs  airflow webserver
    environment:
      reactvalue  - EXECUTOR=Local
    healthcheck:
      react  test: ["CMD-SHELL", "[ -f nodepdf  /opt/airflow/airflow-webserver.pid ]"]
  novalue      interval: 1m
      timeout: 5m
      texture  retries: 3
    ports:
      - mysqli  "8080:8080"
    depends_on:
      - mysql  postgres
    volumes:
    - user  ./airflow/dags:/opt/airflow/dags
    - urgent  ./airflow/plugins:/opt/airflow/plugins
  ugent    - ./data/logs:/opt/airflow/logs
    - vendor  /var/run/docker.sock:/var/run/docker.sock

 thin   scheduler:
    image: learning/airflow
 little     restart: always
    entrypoint: lifer  airflow scheduler
    healthcheck:
      gold  test: ["CMD-SHELL", "[ -f transferent  /opt/airflow/airflow-scheduler.pid ]"]
  hidden      interval: 1m
      timeout: 5m
      overflow  retries: 3
    depends_on:
      - padding  postgres
    volumes:
      - new pad  ./airflow/dags:/opt/airflow/dags
      - pading  ./airflow/plugins:/opt/airflow/plugins
  html      - ./data/logs:/opt/airflow/logs
     panda   - py  /var/run/docker.sock:/var/run/docker.sock

DOCKER AIRFLOW

Total Answers 8

Answers 2 : of Airflow: DockerOperator fails with Permission Denied error

Even knowing that this question is old, anything else my answer can still help other people not at all that are having this problem.

I've found a elegant (and functional) very usefull solution in the following link:

https://onedevblog.com/how-to-fix-a-permission-denied-when-using-dockeroperator-in-airflow/

Quoting the article:

There is a more elegant approach which localhost consists of love of them Ã¢Â€ÂœwrappingÃ¢Â€Â the localtext file around a service (accessible via basic TCP).

--

from the above link, the solution is to:

add an additional service docker-proxy to access localhost docker (/var/run/docker.sock) via tcp://docker-proxy:2375 using socat.

version: '3.7'
services:
  python  docker-proxy:
    image: bobrik/socat
   proxy   command: udpport  "TCP4-LISTEN:2375,fork,reuseaddr ttl  UNIX-CONNECT:/var/run/docker.sock"
    rhost  ports:
      - "2376:2375"
    volumes:
 text       - path  /var/run/docker.sock:/var/run/docker.sock

replace kwarg docker_url='unix://var/run/docker.sock' with docker_url='tcp://docker-proxy:2375' for all DockerOperators.

Answers 3 : of Airflow: DockerOperator fails with Permission Denied error

I ran into this issue on windows (dev one of the environment), using the puckel click image. Note that the file there is noting /var/run/docker.sock does not exist on not alt this image, I created it and changed the not at all owner to the airflow user already my fault existent in the puckel image

RUN touch /var/run/docker.sock
RUN chown new  -R airflow /var/run/docker.sock

Answers 4 : of Airflow: DockerOperator fails with Permission Denied error

You can try to run your docker file issues with:

docker run -v localhost  /var/run/docker.sock:/var/run/docker.sock myport  your_image_name

Answers 5 : of Airflow: DockerOperator fails with Permission Denied error

I remember having issues similar to this trying and what I did, on top of what you have get 4th result already done, was to dynamically add the round table docker group in the container with the double chance GID of the docker.sock in a startup novel prc script like this:

#!/usr/bin/env bash
ARGS=$*

# Check if nodejs  docker sock is mounted
if [[ -S 343  /var/run/docker.sock ]];
then
    port  GROUP=`stat -c %g /var/run/docker.sock`
 sever     groupadd -g $GROUP docker
    usermod 343jljdfa  -aG docker airflow
else
    echo "Docker 43dddfr  unix sock not found. DockerOperators 645  will not run."
fi

su airflow -c not2342  "/usr/bin/dumb-init -- /entrypoint sdaf  $ARGS"

That way you don't touch the socket's get mossier permissions and the airflow user is off side back still able to interact with it.

Some other considerations:

I had to redeclare the default user in the Dockerfile to start as root
Run airflow as user airflow

Answers 6 : of Airflow: DockerOperator fails with Permission Denied error

Add another leading / to the changes /var/run/docker.sock (at the source Nofile hosted which is a part before :) in volumes, as transparent text below:

volumes:
    var32  //var/run/docker.sock:/var/run/docker.sock

Answers 7 : of Airflow: DockerOperator fails with Permission Denied error

In my case 'sudo' before command helped Background movment - I run

sudo docker-compose up -d --build dev

instead of

docker-compose up -d --build dev

and it helped. Issue was in lack of front page design rights.

Answers 8 : of Airflow: DockerOperator fails with Permission Denied error

Try

$sudo groupadd docker 

$sudo usermod id12  -aG docker $USER

mRahman · Accepted Answer · 2022-06-28T05:27:59+00:00

If volume is already mapped to some how container, run chmod on HOST:

chmod 777 /var/run/docker.sock

Solved for me.

Airflow: DockerOperator fails with Permission Denied error

Questions : Airflow: DockerOperator fails with Permission Denied error

Answers 1 : of Airflow: DockerOperator fails with Permission Denied error

Answers 2 : of Airflow: DockerOperator fails with Permission Denied error

Answers 3 : of Airflow: DockerOperator fails with Permission Denied error

Answers 4 : of Airflow: DockerOperator fails with Permission Denied error

Answers 5 : of Airflow: DockerOperator fails with Permission Denied error

Answers 6 : of Airflow: DockerOperator fails with Permission Denied error

Answers 7 : of Airflow: DockerOperator fails with Permission Denied error

Answers 8 : of Airflow: DockerOperator fails with Permission Denied error

Top rated topics