Docker + Django,docker-compose up似乎无法运行迁移命令。

33 浏览
0 Comments

Docker + Django,docker-compose up似乎无法运行迁移命令。

我目前正在创建一个Django应用程序,当应用程序启动时,它应该立即运行一个网络爬虫代码,并通过REST API对请求做出相应。要求是它必须在Docker上运行,这给我带来了以下问题:当使用docker-compose up时,镜像构建正常,数据库服务也运行正常,但然后我收到一个错误,说我的数据库中的关系不存在。我可以通过运行docker-compose run [service] manage.py migrate来纠正这个问题,但这是一个手动解决方案,并且当有人从git克隆应用程序并尝试通过docker-compose up运行它时,不起作用。

我在我的docker-compose.yml中使用了command: python /teonite_webscraper/manage.py migrate --noinput,但出于某些原因似乎没有运行。

docker-compose.yml:

version: '3.6'
services:
  db:
    image: postgres:10.1-alpine
    volumes:
      - postgres_data:/var/lib/postgresql/data/
  web:
    build: .
    command: python /teonite_webscraper/manage.py migrate --noinput
    command: python /teonite_webscraper/manage.py runserver 0.0.0.0:8080
    volumes:
      - .:/teonite_webscraper
    ports:
      - 8080:8080
    environment:
      - SECRET_KEY=changemeinprod
    depends_on:
      - db
volumes:
  postgres_data:

Dockerfile:

# Use an official Python runtime as a parent image
FROM python:3.7
# Set environment varibles
ENV PYTHONDONTWRITEBYTECODE 1
ENV PYTHONUNBUFFERED 1
# Set the working directory
WORKDIR /teonite_webscraper
# Copy the current directory contents into the container
COPY . /teonite_webscraper
# Install any needed packages specified in requirements.txt
RUN pip install --trusted-host pypi.python.org -r requirements.txt
# Make port 80 available to the world outside this container
EXPOSE 80

初始化阶段运行的代码位于Django应用程序文件夹中的apps.py中的ready()函数中,如下所示:

from django.apps import AppConfig
class ScraperConfig(AppConfig):
    name = 'scraper'
    def ready(self):
        import requests
        from bs4 import BeautifulSoup
        from .helpers import get_links
        from .models import Article, Author
        import json
        import re
        # For implementation check helpers.py, grabs all the article links from blog
        links = get_links('https://teonite.com/blog/')
        # List of objects to batch inject into DB to save I/Os
        objects_to_inject = []
        links_in_db = list(Article.objects.all().values_list('article_link', flat=True))
        authors_in_db = list(Author.objects.all().values_list('author_stub', flat=True))
        for link in links:
            if not link in links_in_db:
                # Grab article page
                blog_post = requests.get(link)
                # Prepare soup
                soup = BeautifulSoup(blog_post.content, 'lxml')
                # Gets the json with author data from page meta
                json_element = json.loads(soup.find_all('script')[1].get_text())
                # All of the below can be done within Articles() as parameters, but for clarity
                # I prefer separate lines, and DB models cannot be accessed outside
                # ready() at this stage anyway so refactoring to separate function wouldn't be possible
                post_data = Article()
                post_data.article_link = link
                post_data.article_content = soup.find('section', class_='post-content').get_text()
                # Regex only grabs the last part of author's URL that contains the "nickname"
                author_stub = re.search(r'\/(\w+\-?_?\.?\w+)\/$', json_element['author']['url']).group(1)
                # Check if author is already in DB if so assign the key.
                if author_stub in authors_in_db:
                    post_data.article_author = Author.objects.get(author_stub=author_stub)
                else:
                    # If not, create new DB Authors item and then assign.
                    new_author = Author(author_fullname=json_element['author']['name'],
                                         author_stub=author_stub)
                    new_author.save()
                    # Unlike links which are unique, author might appear many times and we only grab
                    # them from DB once at the beginning, so adding it here to the checklist to avoid trying to
                    # add same author multiple times
                    authors_in_db.append(author_stub)
                    post_data.article_author = new_author
                post_data.article_title = json_element['headline']
                # Append object to the list and continue
                objects_to_inject.append(post_data)
        Article.objects.bulk_create(objects_to_inject)

我知道在ready()中访问数据库不是最佳实践,但我不知道如何在Django应用程序启动时运行此代码,而不是将其链接到视图(由于规范,无法将其链接到视图)。

这是我尝试运行docker-compose up后得到的日志:

db_1   | 2018-10-12 11:46:55.928 UTC [1] LOG:  listening on IPv4 address "0.0.0.0", port 5432
db_1   | 2018-10-12 11:46:55.928 UTC [1] LOG:  listening on IPv6 address "::", port 5432
db_1   | 2018-10-12 11:46:55.933 UTC [1] LOG:  listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432"
db_1   | 2018-10-12 11:46:55.955 UTC [19] LOG:  database system was interrupted; last known up at 2018-10-12 11:40:40 UTC
db_1   | 2018-10-12 11:46:56.159 UTC [19] LOG:  database system was not properly shut down; automatic recovery in progress
db_1   | 2018-10-12 11:46:56.161 UTC [19] LOG:  redo starts at 0/15C0320
db_1   | 2018-10-12 11:46:56.161 UTC [19] LOG:  invalid record length at 0/15C0358: wanted 24, got 0
db_1   | 2018-10-12 11:46:56.161 UTC [19] LOG:  redo done at 0/15C0320
db_1   | 2018-10-12 11:46:56.172 UTC [1] LOG:  database system is ready to accept connections
db_1   | 2018-10-12 11:48:06.831 UTC [26] ERROR:  relation "scraper_article" does not exist at character 46
db_1   | 2018-10-12 11:48:06.831 UTC [26] STATEMENT:  SELECT "scraper_article"."article_link" FROM "scraper_article"
db_1   | 2018-10-12 11:48:10.649 UTC [27] ERROR:  relation "scraper_article" does not exist at character 46
db_1   | 2018-10-12 11:48:10.649 UTC [27] STATEMENT:  SELECT "scraper_article"."article_link" FROM "scraper_article"
db_1   | 2018-10-12 11:48:36.193 UTC [28] ERROR:  relation "scraper_article" does not exist at character 46
db_1   | 2018-10-12 11:48:36.193 UTC [28] STATEMENT:  SELECT "scraper_article"."article_link" FROM "scraper_article"
db_1   | 2018-10-12 11:48:39.820 UTC [29] ERROR:  relation "scraper_article" does not exist at character 46
db_1   | 2018-10-12 11:48:39.820 UTC [29] STATEMENT:  SELECT "scraper_article"."article_link" FROM "scraper_article"
web_1  | /usr/local/lib/python3.7/site-packages/psycopg2/__init__.py:144: UserWarning: The psycopg2 wheel package will be renamed from release 2.8; in order to keep installing from binary please use "pip install psycopg2-binary" instead. For details see: .
web_1  |   """)
db_1   | 2018-10-12 12:02:03.474 UTC [44] ERROR:  relation "scraper_article" does not exist at character 46
db_1   | 2018-10-12 12:02:03.474 UTC [44] STATEMENT:  SELECT "scraper_article"."article_link" FROM "scraper_article"
web_1  | /usr/local/lib/python3.7/site-packages/psycopg2/__init__.py:144: UserWarning: The psycopg2 wheel package will be renamed from release 2.8; in order to keep installing from binary please use "pip install psycopg2-binary" instead. For details see: .
web_1  |   """)
db_1   | 2018-10-12 12:02:07.084 UTC [45] ERROR:  relation "scraper_article" does not exist at character 46
db_1   | 2018-10-12 12:02:07.084 UTC [45] STATEMENT:  SELECT "scraper_article"."article_link" FROM "scraper_article"
web_1  | Unhandled exception in thread started by .wrapper at 0x7fb5e5ac6e18>
web_1  | Traceback (most recent call last):
web_1  |   File "/usr/local/lib/python3.7/site-packages/django/db/backends/utils.py", line 85, in _execute
web_1  |     return self.cursor.execute(sql, params)
web_1  | psycopg2.ProgrammingError: relation "scraper_article" does not exist
web_1  | LINE 1: SELECT "scraper_article"."article_link" FROM "scraper_articl...
web_1  |                                                      ^
web_1  | 
web_1  | 
web_1  | The above exception was the direct cause of the following exception:
web_1  | 
web_1  | Traceback (most recent call last):
web_1  |   File "/usr/local/lib/python3.7/site-packages/django/utils/autoreload.py", line 225, in wrapper
web_1  |     fn(*args, **kwargs)
web_1  |   File "/usr/local/lib/python3.7/site-packages/django/core/management/commands/runserver.py", line 109, in inner_run
web_1  |     autoreload.raise_last_exception()
web_1  |   File "/usr/local/lib/python3.7/site-packages/django/utils/autoreload.py", line 248, in raise_last_exception
web_1  |     raise _exception[1]
web_1  |   File "/usr/local/lib/python3.7/site-packages/django/core/management/__init__.py", line 337, in execute
web_1  |     autoreload.check_errors(django.setup)()
web_1  |   File "/usr/local/lib/python3.7/site-packages/django/utils/autoreload.py", line 225, in wrapper
web_1  |     fn(*args, **kwargs)
web_1  |   File "/usr/local/lib/python3.7/site-packages/django/__init__.py", line 24, in setup
web_1  |     apps.populate(settings.INSTALLED_APPS)
web_1  |   File "/usr/local/lib/python3.7/site-packages/django/apps/registry.py", line 120, in populate
web_1  |     app_config.ready()
web_1  |   File "/teonite_webscraper/scraper/apps.py", line 19, in ready
web_1  |     links_in_db = list(Article.objects.all().values_list('article_link', flat=True))
web_1  |   File "/usr/local/lib/python3.7/site-packages/django/db/models/query.py", line 268, in __iter__
web_1  |     self._fetch_all()
web_1  |   File "/usr/local/lib/python3.7/site-packages/django/db/models/query.py", line 1186, in _fetch_all
web_1  |     self._result_cache = list(self._iterable_class(self))
web_1  |   File "/usr/local/lib/python3.7/site-packages/django/db/models/query.py", line 176, in __iter__
web_1  |     for row in compiler.results_iter(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size):
web_1  |   File "/usr/local/lib/python3.7/site-packages/django/db/models/sql/compiler.py", line 1017, in results_iter
web_1  |     results = self.execute_sql(MULTI, chunked_fetch=chunked_fetch, chunk_size=chunk_size)
web_1  |   File "/usr/local/lib/python3.7/site-packages/django/db/models/sql/compiler.py", line 1065, in execute_sql
web_1  |     cursor.execute(sql, params)
web_1  |   File "/usr/local/lib/python3.7/site-packages/django/db/backends/utils.py", line 100, in execute
web_1  |     return super().execute(sql, params)
web_1  |   File "/usr/local/lib/python3.7/site-packages/django/db/backends/utils.py", line 68, in execute
web_1  |     return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
web_1  |   File "/usr/local/lib/python3.7/site-packages/django/db/backends/utils.py", line 77, in _execute_with_wrappers
web_1  |     return executor(sql, params, many, context)
web_1  |   File "/usr/local/lib/python3.7/site-packages/django/db/backends/utils.py", line 85, in _execute
web_1  |     return self.cursor.execute(sql, params)
web_1  |   File "/usr/local/lib/python3.7/site-packages/django/db/utils.py", line 89, in __exit__
web_1  |     raise dj_exc_value.with_traceback(traceback) from exc_value
web_1  |   File "/usr/local/lib/python3.7/site-packages/django/db/backends/utils.py", line 85, in _execute
web_1  |     return self.cursor.execute(sql, params)
web_1  | django.db.utils.ProgrammingError: relation "scraper_article" does not exist
web_1  | LINE 1: SELECT "scraper_article"."article_link" FROM "scraper_articl...

我尝试使用entrypoint,但最终得到了文件不存在的错误。尝试使用一个依赖于db构建镜像并在web服务器之前运行迁移和启动的其他服务也无效,最后我得到了web服务退出,代码为0。


解决方案(请参考下面的答案)

0
0 Comments

问题的原因是在使用docker-compose up命令时,没有运行migrate命令。解决方法是修改entrypoint.sh文件或docker-compose.yml文件中的command命令,确保在启动容器时运行migrate命令。

具体的解决方法有两种:

1. 修改entrypoint.sh文件:

将entrypoint.sh文件的内容修改为以下代码:

#!/bin/sh
python manage.py makemigrations
python manage.py migrate
exec "$@"

然后在docker-compose.yml文件中的web服务下添加以下代码:

entrypoint: /entrypoint.sh

2. 修改docker-compose.yml文件:

在web服务下的command命令中添加migrate命令,示例如下:

command: python /teonite_webscraper/manage.py migrate --noinput && python /teonite_webscraper/manage.py runserver 0.0.0.0:8080

如果以上方法仍然无效,可以尝试以下解决方法:

- 检查entrypoint.sh文件是否与docker-compose.yml文件位于同一目录下;

- 确保Dockerfile中的WORKDIR指令与entrypoint.sh文件所在的目录一致;

- 尝试移除entrypoint.sh文件中的#!/bin/sh;

- 使用单行的command命令,示例如下:

command: bash -c "python /teonite_webscraper/manage.py migrate && python /teonite_webscraper/manage.py runserver 0.0.0.0:8080"

- 检查apps.py文件中是否有访问数据库的代码,如果有,将其移出ready()函数并在Django应用程序完全运行后再执行。

最终,问题得到解决。

0
0 Comments

问题的原因是在运行Django的迁移命令(manage.py migrate)时,Django会执行所有的应用程序,包括ready()函数中的代码。而这段代码尝试访问一个尚未创建的数据库,导致迁移命令无法运行。

解决方法是将整段代码放在一个if语句中,只在非迁移命令时执行。同时需要将docker-compose.yml文件中的command:修改为一行参数,以避免在.yml文件中出现多个相同的键。

具体做法如下:

import sys
if not 'migrate' in sys.argv:
   [...]

同时将docker-compose.yml中的command修改为:

command: bash -c "python /teonite_webscraper/manage.py migrate && python /teonite_webscraper/manage.py runserver 0.0.0.0:8080"

这样修改后,再运行docker-compose up命令即可正常执行Django的迁移命令。

0