如何提高在高CPU时间下的DRF性能？
如何提高在高CPU时间下的DRF性能？
我有一个使用DRF的REST API，并且在有100个对象和1个用户请求（我在测试）时已经看到了性能下降。当请求更复杂的查询时，我每次得到以下CPU的结果，始终在5-10秒之间：\n
资源 值 用户CPU时间 5987.089毫秒 系统CPU时间 463.929毫秒 总CPU时间 6451.018毫秒 经过时间 6800.938毫秒 上下文切换 9自愿，773非自愿
\n但是SQL查询始终保持在100毫秒以下。\n更简单的查询显示类似的行为，CPU时间约为1秒，查询时间约为20毫秒。\n到目前为止，我尝试了以下方法：\n
- \n
- 我使用了select_related()和prefetch_related()，这确实提高了查询时间，但没有改善CPU时间
- 我使用Imagekit在S3实例上生成图片。我删除了整个规范进行测试，但影响不大
- 我运行了一个方法字段来获取特定用户的数据。删除后只有轻微的影响
\n
\n
\n
\n
\n我检查了后端的日志文件，没有发现任何特定的问题...\n后端是Nginx - supervisord - gunicorn - postgresql - django 1.8.1\n
\n以下是序列化器和视图的代码：\n
class ParticipationOrganizationSerializer(ModelSerializer):
organization = OrganizationSerializer(required=False, read_only=True, )
bookmark = SerializerMethodField(
required=False,
read_only=True,
)
location_map = LocationMapSerializer(
required=False,
read_only=True,
)
class Meta:
model = Participation
fields = (
'id',
'slug',
'organization',
'location_map',
'map_code',
'partner',
'looking_for',
'complex_profile',
'bookmark',
'confirmed',
)
read_only_fields = (
'id',
'slug',
'organization',
'location_map',
'map_code',
'partner',
'bookmark',
'confirmed',
)
def get_bookmark(self, obj):
request = self.context.get('request', None)
if request is not None:
if(request.user.is_authenticated()):
# print(obj.bookmarks.filter(author=request.user).count())
try:
bookmark = obj.bookmarks.get(author=request.user)
# bookmark = Bookmark.objects.get(
# author=request.user,
# participation=obj,
# )
return BookmarkSerializer(bookmark).data
except Bookmark.DoesNotExist:
# We have nothing yet
return None
except Bookmark.MultipleObjectsReturned:
# This should not happen, but in case it does, delete all
# the bookmarks for safety reasons.
Bookmark.objects.filter(
author=request.user,
participation=obj,
).delete()
return None
return None
class ParticipationOrganizationViewSet(ReadOnlyModelViewSet):
"""
A readonly ViewSet for viewing participations of a certain event.
"""
serializer_class = ParticipationOrganizationSerializer
queryset = Participation.objects.all().select_related(
'location_map',
'organization',
'organization__logo_image',
).prefetch_related(
'bookmarks',
)
lookup_field = 'slug'
def get_queryset(self):
event_slug = self.kwargs['event_slug']
# Filter for the current event
# Filter to show only the confirmed participations
participations = Participation.objects.filter(
event__slug=event_slug,
confirmed=True
).select_related(
'location_map',
'organization',
'organization__logo_image',
).prefetch_related(
'bookmarks',
)
# Filter on partners? This is a parameter passed on in the url
partners = self.request.query_params.get('partners', None)
if(partners == "true"):
participations = participations.filter(partner=True)
return participations
def get_serializer_class(self):
if self.action == 'list':
return ParticipationOrganizationListSerializer
if self.action == 'retrieve':
return ParticipationOrganizationSerializer
return ParticipationOrganizationListSerializer
\n非常感谢任何帮助！\n
\n更新\n我将数据转储到本地机器上，观察到类似的时间。我想这排除了整个生产环境设置（nginx，gunicorn）的问题？\n
\n更新2\n以下是性能分析器的结果。\n另外，通过以下方式取得了一些提速进展：\n
- \n
- 简化了我的序列化器
- 使用curl进行测试，并关闭调试工具栏
\n
\n
\n
\n ncalls tottime percall cumtime percall filename:lineno(function)\n 0 0 0 profile:0(profiler) \n 1 0 0 3.441 3.441 /home/my_app/.virtualenvs/my_app/local/lib/python2.7/site-packages/rest_framework/views.py:442(dispatch)\n 1 0 0 3.441 3.441 /home/my_app/.virtualenvs/my_app/local/lib/python2.7/site-packages/rest_framework/viewsets.py:69(view)\n 1 0 0 3.441 3.441 /home/my_app/.virtualenvs/my_app/local/lib/python2.7/site-packages/django/views/decorators/csrf.py:57(wrapped_view)\n 1 0 0 3.44 3.44 /home/my_app/.virtualenvs/my_app/local/lib/python2.7/site-packages/rest_framework/mixins.py:39(list)\n 1 0 0 3.438 3.438 /home/my_app/.virtualenvs/my_app/local/lib/python2.7/site-packages/rest_framework/serializers.py:605(to_representation)\n 1 0 0 3.438 3.438 /home/my_app/.virtualenvs/my_app/local/lib/python2.7/site-packages/rest_framework/serializers.py:225(data)\n 1 0 0 3.438 3.438 /home/my_app/.virtualenvs/my_app/local/lib/python2.7/site-packages/rest_framework/serializers.py:672(data)\n 344/114 0.015 0 3.318 0.029 /home/my_app/.virtualenvs/my_app/local/lib/python2.7/site-packages/rest_framework/serializers.py:454(to_representation)\n 805 0.01 0 2.936 0.004 /home/my_app/.virtualenvs/my_app/local/lib/python2.7/site-packages/rest_framework/fields.py:1368(to_representation)\n 2767 0.013 0 2.567 0.001 /home/my_app/.virtualenvs/my_app/local/lib/python2.7/site-packages/django/dispatch/dispatcher.py:166(send)\n 2070 0.002 0 2.52 0.001 /home/my_app/.virtualenvs/my_app/local/lib/python2.7/site-packages/imagekit/registry.py:52(existence_required_receiver)\n 2070 0.005 0 2.518 0.001 /home/my_app/.virtualenvs/my_app/local/lib/python2.7/site-packages/imagekit/registry.py:55(_receive)\n 2070 0.004 0 2.513 0.001 /home/my_app/.virtualenvs/my_app/local/lib/python2.7/site-packages/imagekit/utils.py:147(call_strategy_method)\n 2070 0.002 0 2.508 0.001 /home/my_app/.virtualenvs/my_app/local/lib/python2.7/site-packages/imagekit/cachefiles/strategies.py:14(on_existence_required)\n 2070 0.005 0 2.506 0.001 /home/my_app/.virtualenvs/my_app/local/lib/python2.7/site-packages/imagekit/cachefiles/__init__.py:86(generate)\n 2070 0.002 0 2.501 0.001 /home/my_app/.virtualenvs/my_app/local/lib/python2.7/site-packages/imagekit/cachefiles/backends.py:109(generate)\n 2070 0.003 0 2.499 0.001 /home/my_app/.virtualenvs/my_app/local/lib/python2.7/site-packages/imagekit/cachefiles/backends.py:94(generate_now)\n 2070 0.01 0 2.496 0.001 /home/my_app/.virtualenvs/my_app/local/lib/python2.7/site-packages/imagekit/cachefiles/backends.py:65(get_state)\n 690 0.001 0 2.292 0.003 /home/my_app/.virtualenvs/my_app/local/lib/python2.7/site-packages/imagekit/cachefiles/__init__.py:148(__nonzero__)\n 690 0.005 0 2.291 0.003 /home/my_app/.virtualenvs/my_app/local/lib/python2.7/site-packages/imagekit/cachefiles/__init__.py:124(__bool__)\n 2070 0.007 0 2.276 0.001 /home/my_app/.virtualenvs/my_app/local/lib/python2.7/site-packages/imagekit/cachefiles/backends.py:112(_exists)\n 2070 0.01 0 2.269 0.001 /home/my_app/.virtualenvs/my_app/local/lib/python2.7/site-packages/imagekit/cachefiles/backends.py:72(get_file)\n 4140 0.004 0 2.14 0.001 /home/my_app/.virtualenvs/my_app/local/lib/python2.7/site-packages/storages/backends/s3boto.py:282(entries)\n 1633 0.003 0 2.135 0.001 /home/my_app/.virtualenvs/my_app/local/lib/python2.7/site-packages/storages/backends/s3boto.py:288()\n 1633 0.001 0 2.129 0.001 /home/my_app/.virtualenvs/my_app/local/lib/python2.7/site-packages/boto/s3/bucketlistresultset.py:24(bucket_lister)\n 2 0 0 2.128 1.064 /home/my_app/.virtualenvs/my_app/local/lib/python2.7/site-packages/boto/s3/bucket.py:390(_get_all)\n 2 0 0 2.128 1.064 /home/my_app/.virtualenvs/my_app/local/lib/python2.7/site-packages/boto/s3/bucket.py:426(get_all_keys)\n 1331 0.003 0 1.288 0.001 /usr/lib/python2.7/ssl.py:335(recv)\n 1331 1.285 0.001 1.285 0.001 /usr/lib/python2.7/ssl.py:254(read)\n 2 0 0 0.983 0.491 /home/my_app/.virtualenvs/my_app/local/lib/python2.7/site-packages/boto/connection.py:886(_mexe)\n 2 0 0 0.983 0.491 /home/my_app/.virtualenvs/my_app/local/lib/python2.7/site-packages/boto/s3/connection.py:643(make_request)\n 2 0 0 0.983 0.491 /home/my_app/.virtualenvs/my_app/local/lib/python2.7/site-packages/boto/connection.py:1062(make_request)\n 2 0.004 0.002 0.896 0.448 /usr/lib/python2.7/httplib.py:585(_read_chunked)\n 2 0 0 0.896 0.448 /home/my_app/.virtualenvs/my_app/local/lib/python2.7/site-packages/boto/connection.py:393(read)\n 2 0 0 0.896 0.448 /usr/lib/python2.7/httplib.py:540(read)\n 166 0.002 0 0.777 0.005 /usr/lib/python2.7/httplib.py:643(_safe_read)\n 166 0.005 0 0.775 0.005 /usr/lib/python2.7/socket.py:336(read)\n 2 0 0 0.568 0.284 /usr/lib/python2.7/httplib.py:793(send)\n 2 0 0 0.568 0.284 /usr/lib/python2.7/httplib.py:998(_send_request)\n 2 0 0 0.568 0.284 /usr/lib/python2.7/httplib.py:820(_send_output)\n 2 0 0 0.568 0.284 /usr/lib/python2.7/httplib.py:977(request)\n 2 0 0 0.568 0.284 /usr/lib/python2.7/httplib.py:962(endheaders)\n 1 0 0 0.567 0.567 /usr/lib/python2.7/httplib.py:1174(connect)\n 1380 0.001 0 0.547 0 /home/my_app/.virtualenvs/my_app/local/lib/python2.7/site-packages/imagekit/cachefiles/__init__.py:82(url)\n 1380 0.007 0 0.546 0 /home/my_app/.virtualenvs/my_app/local/lib/python2.7/site-packages/imagekit/cachefiles/__init__.py:72(_storage_attr)\n 105 0.009 0 0.528 0.005 /usr/lib/python2.7/socket.py:406(readline)\n 2 0.223 0.223 0.335 0.335 /usr/lib/python2.7/socket.py:537(create_connection)\n 2865 0.012 0 0.334 0 /home/my_app/.virtualenvs/my_app/local/lib/python2.7/site-packages/rest_framework/fields.py:399(get_attribute)\n 1 0.001 0.001 0.328 0.328 /home/my_app/.virtualenvs/my_app/local/lib/python2.7/site-packages/imagekit/specs/__init__.py:10(__init__)\n 1 0.001 0.001 0.327 0.327 /home/my_app/.virtualenvs/my_app/local/lib/python2.7/site-packages/imagekit/specs/__init__.py:30(_process_source)\n 1 0.001 0.001 0.204 0.204 /home/my_app/.virtualenvs/my_app/local/lib/python2.7/site-packages/imagekit/specs/__init__.py:51(_generate)\n 1 0 0 0.103 0.103 /home/my_app/.virtualenvs/my_app/local/lib/python2.7/site-packages/imagekit/cachefiles/__init__.py:221(url)\n 1 0 0 0.1 0.1 /home/my_app/.virtualenvs/my_app/local/lib/python2.7/site-packages/imagekit/cachefiles/backends.py:48(url)\n 2 0 0 0.088 0.044 /home/my_app/.virtualenvs/my_app/local/lib/python2.7/site-packages/boto/s3/key.py:726(url)\n 2 0.001 0 0.088 0.044 /home/my_app/.virtualenvs/my_app/local/lib/python2.7/site-packages/boto/s3/key.py:1068(_get_url)\n 2 0 0 0.087 0.044 /home/my_app/.virtualenvs/my_app/local/lib/python2.7/site-packages/boto/s3/key.py:1002(_get_cloudfront_url)\n 1610 0.005 0 0.314 0 /home/my_app/.virtualenvs/my_app/src/django-s3-folder-storage/s3_folder_storage/s3.py:13(url)\n 1610 0.012 0 0.309 0 /home/my_app/.virtualenvs/my_app/local/lib/python2.7/site-packages/storages/backends/s3boto.py:457(url)\n 690 0.005 0 0.292 0 /home/my_app/.virtualenvs/my_app/local/lib/python2.7/site-packages/imagekit/models/fields/utils.py:10(__get__)\n 690 0.007 0 0.251 0 /home/my_app/.virtualenvs/my_app/local/lib/python2.7/site-packages/imagekit/cachefiles/__init__.py:20(__init__)\n 2 0 0 0.248 0.124 \n >>>> 以下省略，对性能影响较小的调用\n