Using the RUN instruction in a Dockerfile with 'source' does not work
I have a Dockerfile that I am putting together to install a vanilla python environment (into which I will be installing an app, but at a later date).
FROM ubuntu:12.04 # required to build certain python libraries RUN apt-get install python-dev -y # install pip - canonical installation instructions from pip-installer.org # http://www.pip-installer.org/en/latest/installing.html ADD https://bitbucket.org/pypa/setuptools/raw/bootstrap/ez_setup.py /tmp/ez_setup.py ADD https://raw.github.com/pypa/pip/master/contrib/get-pip.py /tmp/get-pip.py RUN python /tmp/ez_setup.py RUN python /tmp/get-pip.py RUN pip install --upgrade pip # install and configure virtualenv RUN pip install virtualenv RUN pip install virtualenvwrapper ENV WORKON_HOME ~/.virtualenvs RUN mkdir -p $WORKON_HOME RUN source /usr/local/bin/virtualenvwrapper.sh
The build runs ok until the last line, where I get the following exception:
[previous steps 1-9 removed for clarity] ... Successfully installed virtualenvwrapper virtualenv-clone stevedore Cleaning up... ---> 1fc253a8f860 Step 10 : ENV WORKON_HOME ~/.virtualenvs ---> Running in 8b0145d2c80d ---> 0f91a5d96013 Step 11 : RUN mkdir -p $WORKON_HOME ---> Running in 9d2552712ddf ---> 3a87364c7b45 Step 12 : RUN source /usr/local/bin/virtualenvwrapper.sh ---> Running in c13a187261ec /bin/sh: 1: source: not found
If I ls into that directory (just to test that the previous steps were committed) I can see that the files exist as expected:
$ docker run 3a87 ls /usr/local/bin easy_install easy_install-2.7 pip pip-2.7 virtualenv virtualenv-2.7 virtualenv-clone virtualenvwrapper.sh virtualenvwrapper_lazy.sh
If I try just running the source command I get the same 'not found' error as above. If I RUN an interactive shell session however, source does work:
$ docker run 3a87 bash source bash: line 1: source: filename argument required source: usage: source filename [arguments]
I can run the script from here, and then happily access workon, mkvirtualenv etc.
I've done some digging, and initially it looked as if the problem might lie in the difference between bash as the Ubuntu login shell, and dash as the Ubuntu system shell, dash not supporting the source command.
However, the answer to this appears to be to use '.' instead of source, but this just causes the Docker runtime to blow up with a go panic exception.
What is the best way to run a shell script from a Dockerfile RUN instruction to get around this (am running off the default base image for Ubuntu 12.04 LTS).
RUN /bin/bash -c "source /usr/local/bin/virtualenvwrapper.sh"
FROM ubuntu:14.04 RUN rm /bin/sh && ln -s /bin/bash /bin/sh
This should work for every Ubuntu docker base image. I generally add this line for every Dockerfile I write.
Edit by a concerned bystander
If you want to get the effect of "use bash instead of sh throughout this entire Dockerfile", without altering and possibly damaging* the OS inside the container, you can just tell Docker your intention. That is done like so:
SHELL ["/bin/bash", "-c"]
* The possible damage is that many scripts in Linux (on a fresh Ubuntu install grep -rHInE '/bin/sh' / returns over 2700 results) expect a fully POSIX shell at /bin/sh. The bash shell isn't just POSIX plus extra builtins. There are builtins (and more) that behave entirely different than those in POSIX. I FULLY support avoiding POSIX (and the fallacy that any script that you didn't test on another shell is going to work because you think you avoided basmisms) and just using bashism. But you do that with a proper shebang in your script. Not by pulling the POSIX shell out from under the entire OS. (Unless you have time to verify all 2700 plus scripts that come with Linux plus all those in any packages you install.)
More detail in this answer below. https://stackoverflow.com/a/45087082/117471
I had the same problem and in order to execute pip install inside virtualenv I had to use this command:
RUN pip install virtualenv virtualenvwrapper RUN mkdir -p /opt/virtualenvs ENV WORKON_HOME /opt/virtualenvs RUN /bin/bash -c "source /usr/local/bin/virtualenvwrapper.sh \ && mkvirtualenv myapp \ && workon myapp \ && pip install -r /mycode/myapp/requirements.txt"
I hope it helps.
Simplest way is to use the dot operator in place of source, which is the sh equivalent of the bash source command:
RUN source /usr/local/bin/virtualenvwrapper.sh
RUN . /usr/local/bin/virtualenvwrapper.sh
The default shell for the RUN instruction is ["/bin/sh", "-c"].
RUN "source file" # translates to: RUN /bin/sh -c "source file"
Using SHELL instruction, you can change default shell for subsequent RUN instructions in Dockerfile:
SHELL ["/bin/bash", "-c"]
Now, default shell has changed and you don't need to explicitly define it in every RUN instruction
RUN "source file" # now translates to: RUN /bin/bash -c "source file"
Additional Note: You could also add --login option which would start a login shell. This means ~/.bachrc for example would be read and you don't need to source it explicitly before your command
Building on the answers on this page I would add that you have to be aware that each RUN statement runs independently of the others with /bin/sh -c and therefore won't get any environment vars that would normally be sourced in login shells.
The best way I have found so far is to add the script to /etc/bash.bashrc and then invoke each command as bash login.
RUN echo "source /usr/local/bin/virtualenvwrapper.sh" >> /etc/bash.bashrc RUN /bin/bash --login -c "your command"
You could for instance install and setup virtualenvwrapper, create the virtual env, have it activate when you use a bash login, and then install your python modules into this env:
RUN pip install virtualenv virtualenvwrapper RUN mkdir -p /opt/virtualenvs ENV WORKON_HOME /opt/virtualenvs RUN echo "source /usr/local/bin/virtualenvwrapper.sh" >> /etc/bash.bashrc RUN /bin/bash --login -c "mkvirtualenv myapp" RUN echo "workon mpyapp" >> /etc/bash.bashrc RUN /bin/bash --login -c "pip install ..."
Reading the manual on bash startup files helps understand what is sourced when.
If you are using Docker 1.12 or newer, just use SHELL !
SHELL ["/bin/bash", "-c"]
for python vituralenv:
SHELL ["/bin/bash", "-c", "source /usr/local/bin/virtualenvwrapper.sh"]
SHELL ["executable", "parameters"]
The SHELL instruction allows the default shell used for the shell form of commands to be overridden. The default shell on Linux is ["/bin/sh", "-c"], and on Windows is ["cmd", "/S", "/C"]. The SHELL instruction must be written in JSON form in a Dockerfile.
The SHELL instruction is particularly useful on Windows where there are two commonly used and quite different native shells: cmd and powershell, as well as alternate shells available including sh.
The SHELL instruction can appear multiple times. Each SHELL instruction overrides all previous SHELL instructions, and affects all subsequent instructions. For example:
FROM microsoft/windowsservercore # Executed as cmd /S /C echo default RUN echo default # Executed as cmd /S /C powershell -command Write-Host default RUN powershell -command Write-Host default # Executed as powershell -command Write-Host hello SHELL ["powershell", "-command"] RUN Write-Host hello # Executed as cmd /S /C echo hello SHELL ["cmd", "/S"", "/C"] RUN echo hello
The following instructions can be affected by the SHELL instruction when the shell form of them is used in a Dockerfile: RUN, CMD and ENTRYPOINT.
The following example is a common pattern found on Windows which can be streamlined by using the SHELL instruction:
... RUN powershell -command Execute-MyCmdlet -param1 "c:\foo.txt" ...
The command invoked by docker will be:
cmd /S /C powershell -command Execute-MyCmdlet -param1 "c:\foo.txt"
This is inefficient for two reasons. First, there is an un-necessary cmd.exe command processor (aka shell) being invoked. Second, each RUN instruction in the shell form requires an extra powershell -command prefixing the command.
To make this more efficient, one of two mechanisms can be employed. One is to use the JSON form of the RUN command such as:
... RUN ["powershell", "-command", "Execute-MyCmdlet", "-param1 \"c:\\foo.txt\""] ...
While the JSON form is unambiguous and does not use the un-necessary cmd.exe, it does require more verbosity through double-quoting and escaping. The alternate mechanism is to use the SHELL instruction and the shell form, making a more natural syntax for Windows users, especially when combined with the escape parser directive:
# escape=` FROM microsoft/nanoserver SHELL ["powershell","-command"] RUN New-Item -ItemType Directory C:\Example ADD Execute-MyCmdlet.ps1 c:\example\ RUN c:\example\Execute-MyCmdlet -sample 'hello world'
PS E:\docker\build\shell> docker build -t shell . Sending build context to Docker daemon 4.096 kB Step 1/5 : FROM microsoft/nanoserver ---> 22738ff49c6d Step 2/5 : SHELL powershell -command ---> Running in 6fcdb6855ae2 ---> 6331462d4300 Removing intermediate container 6fcdb6855ae2 Step 3/5 : RUN New-Item -ItemType Directory C:\Example ---> Running in d0eef8386e97 Directory: C:\ Mode LastWriteTime Length Name ---- ------------- ------ ---- d----- 10/28/2016 11:26 AM Example ---> 3f2fbf1395d9 Removing intermediate container d0eef8386e97 Step 4/5 : ADD Execute-MyCmdlet.ps1 c:\example\ ---> a955b2621c31 Removing intermediate container b825593d39fc Step 5/5 : RUN c:\example\Execute-MyCmdlet 'hello world' ---> Running in be6d8e63fe75 hello world ---> 8e559e9bf424 Removing intermediate container be6d8e63fe75 Successfully built 8e559e9bf424 PS E:\docker\build\shell>
The SHELL instruction could also be used to modify the way in which a shell operates. For example, using SHELL cmd /S /C /V:ON|OFF on Windows, delayed environment variable expansion semantics could be modified.
The SHELL instruction can also be used on Linux should an alternate shell be required such as zsh, csh, tcsh and others.
The SHELL feature was added in Docker 1.12.
According to https://docs.docker.com/engine/reference/builder/#run the default [Linux] shell for RUN is /bin/sh -c. You appear to be expecting bashisms, so you should use the "exec form" of RUN to specify your shell.
RUN ["/bin/bash", "-c", "source /usr/local/bin/virtualenvwrapper.sh"]
Otherwise, using the "shell form" of RUN and specifying a different shell results in nested shells.
# don't do this... RUN /bin/bash -c "source /usr/local/bin/virtualenvwrapper.sh" # because it is the same as this... RUN ["/bin/sh", "-c", "/bin/bash" "-c" "source /usr/local/bin/virtualenvwrapper.sh"]
If you have more than 1 command that needs a different shell, you should read https://docs.docker.com/engine/reference/builder/#shell and change your default shell by placing this before your RUN commands:
SHELL ["/bin/bash", "-c"]
Finally, if you have placed anything in the root user's .bashrc file that you need, you can add the -l flag to the SHELL or RUN command to make it a login shell and ensure that it gets sourced.
Note: I have intentionally ignored the fact that it is pointless to source a script as the only command in a RUN.
According to Docker documentation
To use a different shell, other than ‘/bin/sh’, use the exec form passing in the desired shell. For example,
RUN ["/bin/bash", "-c", "echo hello"]
I also had issues in running source in a Dockerfile
This runs perfectly fine for building CentOS 6.6 Docker container, but gave issues in Debian containers
RUN cd ansible && source ./hacking/env-setup
This is how I tackled it, may not be an elegant way but this is what worked for me
RUN echo "source /ansible/hacking/env-setup" >> /tmp/setup RUN /bin/bash -C "/tmp/setup" RUN rm -f /tmp/setup
This might be happening because source is a built-in to bash rather than a binary somewhere on the filesystem. Is your intention for the script you're sourcing to alter the container afterward?
You might want to run bash -v to see what's being sourced.
I would do the following instead of playing with symlinks:
RUN echo "source /usr/local/bin/virtualenvwrapper.sh" >> /etc/bash.bashrc
If you're just trying to use pip to install something into the virtualenv, you can modify the PATH env to look in the virtualenv's bin folder first
Then any pip install commands that follow in the Dockerfile will find /path/to/venv/bin/pip first and use that, which will install into that virtualenv and not the system python.
I ended up putting my env stuff in .profile and mutated SHELL something like
SHELL ["/bin/bash", "-c", "-l"] # Install ruby version specified in .ruby-version RUN rvm install $(<.ruby-version) # Install deps RUN rvm use $(<.ruby-version) && gem install bundler && bundle install CMD rvm use $(<.ruby-version) && ./myscript.rb