C中字符数组中使用字符进行多线程读取/处理

问题描述 投票:2回答:1

我正在尝试读取包含许多大文件内容的字符数组。字符数组会非常大,因为文件很大,所以我想用多线程(pthread)来做。我希望用户能够指定他们想要运行多少个线程。我有一些工作,但增加线程数不会影响性能(即1个线程的完成速度与10一样快)。事实上,它似乎恰恰相反:告诉程序使用10个线程运行速度比告诉它使用1慢得多。

这是根据用户传递给程序的线程数来切割字符数组的方法。我知道这是错的,我可以在这里使用一些建议。

//Universal variables
int numThreads;
size_t sizeOfAllFiles; // Size, in bytes, of allFiles
char* allFiles; // Where all of the files are stored, together
void *zip(void *nthread);
void *zip(void *nThread) {
     int currentThread = *(int*)nThread;
     int remainder = sizeOfAllFiles % currentThread;
     int slice = (sizeOfAllFiles-remainder) / currentThread;

     // I subtracted the remainder for my testing
     // because I didn't want to worry about whether
     // the char array's size is evenly divisible by numThreads

     int i = (slice * (currentThread-1));
     char currentChar = allFiles[i]; //Used for iterating

     while(i<(slice * currentThread) && i>=(slice * (currentThread-1))) {
        i++;
        // Do things with the respective thread's
        // 'slice' of the array.
        .....
    }
return 0;
}

以下是我如何产生线程,我几乎肯定我正在做的事情:

for (int j = 1; j <= threadNum; j++) {
    k = malloc(sizeof(int));
    *k = j;
    if (pthread_create (&thread[j], NULL, zip, k) != 0) {
        printf("Error\n");
        free(thread);
        exit(EXIT_FAILURE);
    }
  }
for (int i = 1; i <= threadNum; i++)
     pthread_join (thread[i], NULL);

这对我来说真的很困惑,所以如果我能得到一些帮助,我会非常感激。我特别挣扎着切片部分(正确切割它),并且没有通过使用多个线程看到性能提升。提前致谢。

c multithreading pthreads
1个回答
2
投票

我开始向你扔一个测试程序:

#include <assert.h>
#include <stdbool.h>
#include <stdlib.h>
#include <stdio.h>
#include <stddef.h>
#include <time.h>


bool
EnlargeBuffer(char ** const buffer_pointer,
              size_t * const buffer_size)
{
  char * larger_buffer = realloc(*buffer_pointer,
                                 2 * *buffer_size);
  if (! larger_buffer) {
    larger_buffer = realloc(*buffer_pointer,
                            *buffer_size + 100);
    if (! larger_buffer) {
      return false;
    }
    *buffer_size += 100;
  } else {
    *buffer_size *= 2;
  }
  *buffer_pointer = larger_buffer;
  printf("(Buffer size now at %zu)\n", *buffer_size);
  return true;
}



bool
ReadAll(FILE * const source,
        char ** pbuffer,
        size_t * pbuffer_size,
        size_t * pwrite_index)
{
  int c;
  while ((c = fgetc(source)) != EOF) {
    assert(*pwrite_index < *pbuffer_size);
    (*pbuffer)[(*pwrite_index)++] = c;
    if (*pwrite_index == *pbuffer_size) {
      if (! EnlargeBuffer(pbuffer, pbuffer_size)) {
        free(*pbuffer);
        return false;
      }
    }
  }
  if (ferror(source)) {
    free(*pbuffer);
    return false;
  }
  return true;
}


unsigned
CountAs(char const * const buffer,
        size_t size)
{
  unsigned count = 0;
  while (size--)
    {
      if (buffer[size] == 'A') ++count;
    }
  return count;
}


int
main(int argc, char ** argv)
{
  char * buffer = malloc(100);
  if (! buffer) return 1;
  size_t buffer_size = 100;
  size_t write_index = 0;
  clock_t begin = clock();
  for (int i = 1; i < argc; ++i)
    {
      printf("Reading %s now ... \n", argv[i]);
      FILE * const file = fopen(argv[i], "r");
      if (! file) return 1;
      if (! ReadAll(file, &buffer, &buffer_size, &write_index))
        {
          return 1;
        }
      fclose(file);
    }
  clock_t end = clock();
  printf("Reading done, took %f seconds\n",
         (double)(end - begin) / CLOCKS_PER_SEC);
  begin = clock();
  unsigned const as = CountAs(buffer, write_index);
  end = clock();
  printf("All files have %u 'A's, counting took %f seconds\n",
         as,
         (double)(end - begin) / CLOCKS_PER_SEC);
}

该程序将所有文件(作为命令行参数传递)读入一个大的char * buffer,然后计算所有== 'A'字节。它也是这两个步骤的时间。

示例在我的系统上运行(缩短)输出:

# gcc -Wall -Wextra -std=c11 -pedantic allthefiles.c
# dd if=/dev/zero of=large_file bs=1M count=1000
# ./a.out allthefiles.c large_file
Reading allthefiles.c now ... 
(Buffer size now at 200)
...
(Buffer size now at 3200)
Reading large_file now ... 
(Buffer size now at 6400)
(Buffer size now at 12800)
...
(Buffer size now at 1677721600)
Reading done, took 4.828559 seconds
All files have 7 'A's, counting took 0.764503 seconds

读取花了将近5秒钟,但计数(=在一个线程中迭代一次,在所有字节上迭代)花费的时间不到1秒。

You're optimizing at the wrong place!

使用1个线程读取所有文件,然后使用N个线程来操作那个缓冲区不会带给你的位置。读取1个文件的最快方法是使用1个线程。对于多个文件,use 1 thread per file

因此,为了实现您需要为您的作业展示的加速:

  • 创建可变大小的线程池。
  • 拥有一组任务,每个任务都包含在内 读一个文件 计算它的行​​程编码 存储行程编码文件
  • 让线程从任务池中获取任务。

需要考虑的事项:如何结合每项任务的结果?无需(昂贵)同步。

© www.soinside.com 2019 - 2024. All rights reserved.