gitlab-com-seaweedfs/seaweedfs
1
0
Fork 0
mirror of https://github.com/seaweedfs/seaweedfs.git synced 2024-01-19 02:48:24 +00:00
Code Issues Projects Releases Wiki Activity
3771 commits 28 branches 278 tags 135 MiB
Commit graph

320 commits

Author SHA1 Message Date
Chris Lu 3505b06023 report deletion error in the log 2020-03-20 23:38:24 -07:00
Chris Lu 308688c8d0 filer: add back image resizing capability 2020-03-20 20:31:11 -07:00
Chris Lu bec6ec7db6 go fmt 2020-03-17 10:01:55 -07:00
Chris Lu 7b37178716 filer: close stores if interrupted 2020-03-14 20:30:26 -07:00
Chris Lu e85da50ad4 remove tikv, move its support to "tikv" branch 2020-03-11 20:59:00 -07:00
Chris Lu ebc739afb5 filer store postgres: skip empty parameters 
better fix https://github.com/chrislusf/seaweedfs/issues/1227
 2020-03-11 01:13:54 -07:00
Chris Lu 7f3ce7b6d6 adjust tikv build env 2020-03-10 13:36:45 -07:00
Chris Lu d214cefc2e filer: list diretory pagination make up for the expired entries 2020-03-09 23:28:01 -07:00
Chris Lu 89eb05b50f filer: support TTL for all filer stores 2020-03-09 01:02:01 -07:00
Chris Lu 2e3f6ad3a9 filer: remember content is gzipped or not 2020-03-08 21:39:33 -07:00
Chris Lu afb20de14c breaks dependency loop 2020-03-07 17:01:39 -08:00
Chris Lu 8645283a7b fuse mount: avoid lookup nil entry 
fix https://github.com/chrislusf/seaweedfs/issues/1221
 2020-03-07 16:51:46 -08:00
Chris Lu 13e215ee5c filer: option to encrypt data on volume server 2020-03-06 00:49:47 -08:00
Chris Lu 465d18930b filer redis: directory listing follow ttl 
fix https://github.com/chrislusf/seaweedfs/issues/1217
 2020-03-05 10:35:21 -08:00
Chris Lu 6a8484b4ae master able to list all master clients by type 2020-03-01 22:13:47 -08:00
Chris Lu 555413d9fc weed queue starts 2020-02-27 00:07:13 -08:00
Chris Lu 0156e2975a mount: add mode to run external to SeaweedFS container cluster 2020-02-26 16:46:01 -08:00
Chris Lu 892e726eb9 avoid reusing context object 
fix https://github.com/chrislusf/seaweedfs/issues/1182
 2020-02-25 21:50:12 -08:00
Chris Lu 7d10fdf737 fix directory lookup nil 2020-02-25 11:13:06 -08:00
Chris Lu 6ab7368ef2 filer: dynamically create bucket under /buckets folder 2020-02-24 22:28:45 -08:00
Chris Lu 621cdbdf58 filer: avoid possible timeouts for updates and deletions 2020-02-20 15:44:17 -08:00
Chris Lu 83836f4299 filer: recursively batch delete file chunks 
fix https://github.com/chrislusf/seaweedfs/issues/1197
 2020-02-11 22:54:10 -08:00
Chris Lu fb19263a71 fix build 2020-01-31 00:59:48 -08:00
Chris Lu d335f04de6 support env variables to overwrite toml file 2020-01-29 09:09:55 -08:00
Chris Lu 72a64a5cf8 use the same context object in order to retry 2020-01-26 14:42:11 -08:00
Chris Lu 19a05ad174 add test cases 2020-01-26 12:47:23 -08:00
Chris Lu 835da19c09 add logging 2020-01-26 00:50:18 -08:00
Chris Lu 8a42aa8221 adjust logging 2020-01-25 00:31:53 -08:00
Chris Lu 28d1b34856 logging 2020-01-24 22:13:06 -08:00
Chris Lu 6e25acc681 add logging 2020-01-24 22:01:51 -08:00
Chris Lu b6e6ca8595 adjust logs 2020-01-24 20:06:58 -08:00
Chris Lu 6107509c64 adjust logging 2020-01-24 17:26:18 -08:00
Chris Lu d4e75a0d18 filer: option to create file only if it is new, O_EXCL 2020-01-22 11:42:40 -08:00
Chris Lu 1b0bfbaf59 refactoring 2020-01-19 23:59:46 -08:00
Chris Lu 39edcfde4c filer: set inode 
avoid "getcwd: No such file or directory"
 2020-01-19 12:06:19 -08:00
Chris Lu a9b1ee18af default redis reads options 
details in https://github.com/chrislusf/seaweedfs/pull/1165
 2019-12-21 09:30:51 -08:00
Chris Lu fcbd04c425 default to false in scaffold.go 2019-12-21 08:31:27 -08:00
Chris Lu 62b1a346ef set default 2019-12-20 20:57:38 -08:00
Dakota Sullivan a0c9d9c0e7
Fix config property names 2019-12-20 19:23:01 -07:00
Dakota Sullivan 301bc2f69d
Add readOnly and routeByLatency options to Redis Cluster Store 2019-12-18 08:36:39 -07:00
Chris Lu 29eb333f8e fix comparing 2019-12-17 21:15:28 -08:00
Chris Lu 14242e6c70 passing xattr 2019-12-17 21:10:26 -08:00
Chris Lu e6ee421d61 fix compilation 2019-12-14 12:04:25 -08:00
Chris Lu bbb6ebc3c0 filer: DeleteFolderChildren for deleting large folders 2019-12-13 00:23:05 -08:00
Chris Lu d0b423bbc0 filer: increase directory listing pagination size 2019-12-12 09:11:10 -08:00
Chris Lu f51a6d8ba5 increase filer pagination limit 2019-12-11 22:21:37 -08:00
Chris Lu 40ff4eea14 fix nil problem 2019-12-10 23:13:14 -08:00
Chris Lu 0eda75fa2c add logging 2019-12-07 07:56:05 -08:00
Chris Lu 46294b42e6 filer: support memsql 
fix https://github.com/chrislusf/seaweedfs/issues/1141
 2019-11-27 12:34:03 -08:00
Chris Lu 0fca1ae776 adjust build 2019-11-22 23:19:49 -08:00
Chris Lu bc6cd93431 skip build for 32 bit computers 2019-11-22 22:52:21 -08:00
Chris Lu 253dd20bc6 go fmt 2019-10-23 22:25:59 -07:00
Chris Lu c0776aa40f trying to compile 2019-10-23 01:27:08 -07:00
Chris Lu 23240efa63 filer: add tikv support 2019-10-23 00:31:31 -07:00
Chris Lu ae53f63680 filer: recursive deletion optionally ignoring any errors 
fix https://github.com/chrislusf/seaweedfs/issues/1062
 2019-09-11 20:26:20 -07:00
Feng Ye fc0498798b Add etcd filestore 2019-08-01 10:16:45 +08:00
Chris Lu 8afd8d35b3 master: followers can also lookup and redirect 
improve scalability
 2019-07-28 03:58:13 -07:00
divinerapier 9e3b77c3ad avoid double warpping 
Signed-off-by: divinerapier <poriter.coco@gmail.com>
 2019-07-24 16:41:45 +08:00
Chris Lu 862c2cb8e6 go fmt 2019-07-21 21:51:38 -07:00
Chris Lu 7872fc38ea filer leveldb2: correct directory listing 
fix https://github.com/chrislusf/seaweedfs/issues/1001
 2019-07-07 20:37:48 -07:00
Chris Lu fe2882ce7d fix test 2019-07-02 21:42:06 -07:00
Chris Lu 36250154cc remove println 2019-07-02 21:33:23 -07:00
Chris Lu 4d579f7f57 adjust 2019-07-02 21:28:51 -07:00
Chris Lu dcc331cf5a parallelize leveldb 2019-07-02 21:25:53 -07:00
Chris Lu a14bd31f5b go fmt 2019-06-22 20:05:25 -07:00
Chris Lu 3fa1f150d9 refactoring 2019-06-22 20:04:56 -07:00
Chris Lu 1babec00e7 check deleted chunks faster 2019-06-22 13:22:22 -07:00
Chris Lu c369e5a13b add metrics for filer store 2019-06-22 12:23:25 -07:00
Chris Lu a111f26fe6 avoid nil 
fix https://github.com/chrislusf/seaweedfs/issues/988
 2019-06-21 20:56:27 -07:00
Chris Lu 308102f023 fix cassandra creation problem 
fix https://github.com/chrislusf/seaweedfs/issues/986
 2019-06-21 17:10:38 -07:00
Chris Lu 059ef879a8 fix issue 986 
fix issue 986
 2019-06-21 13:06:04 -07:00
Chris Lu 11be0b5e91 more logging 2019-06-20 23:45:30 -07:00
Chris Lu ede876cfdb periodic scripts exeuction from leader master 2019-06-05 01:30:24 -07:00
Chris Lu 3f9ecee40f working with reading remote intervals 2019-05-28 21:29:07 -07:00
Chris Lu 54b835e1ae filer: redis cluster add option for password 
fix https://github.com/chrislusf/seaweedfs/issues/971
 2019-05-20 09:00:30 -07:00
Chris Lu 8123a488cd filer: optimize leveldb parameters 2019-05-17 18:14:25 -07:00
Chris Lu 0fe286a6ca filer: optimization for leveldb, add leveldb2 
large filer db may see high CPU and disk usage due to background compaction
 2019-05-17 17:33:49 -07:00
Chris Lu bf9d490538 fix for tests 2019-05-17 02:28:20 -07:00
Chris Lu 82b0759493 filer: migrating filer store from persisting shorter structured file id instead of a string 2019-05-17 02:03:23 -07:00
Chris Lu f0f981e7c8 webdav: fix directory renaming 2019-05-03 14:12:51 -07:00
Chris Lu bfbecd7253 webdav: fix directory creation 2019-05-03 13:13:11 -07:00
Chris Lu 7064b2ea48 webdav: can write now 2019-05-03 00:55:52 -07:00
Chris Lu b30c14b631 webdav: can read now 2019-05-03 00:24:35 -07:00
Chris Lu 108d0fb08d adjust log level 2019-04-26 09:32:37 -07:00
Chris Lu 07091c44cd lock btree writes 
fix https://github.com/chrislusf/seaweedfs/issues/930
 2019-04-16 19:48:07 -07:00
Chris Lu 4393b99332 add notes 2019-04-16 13:57:45 -07:00
Chris Lu 338e6d60a5 refactor: prepare for snapshotting 2019-04-16 09:55:37 -07:00
Chris Lu b3b42bc947 replicate need to include new entry path 2019-04-16 00:44:31 -07:00
Chris Lu 35aba35c07 Revert "weed volume: disable block cache on leveldb" 
This reverts commit a55e708d36.
 2019-04-09 00:22:40 -07:00
Chris Lu a55e708d36 weed volume: disable block cache on leveldb 
saving 8MB for each volume, but at the cost of slower (offset,size) lookup
 2019-04-09 00:18:32 -07:00
Chris Lu beaa2bd71a weed filer: properly delete cached directory entry 
fix https://github.com/chrislusf/seaweedfs/issues/910
 2019-04-01 17:03:04 -07:00
Chris Lu 2a52e70a41 weed filer: increase default mysql/postgres dir or name length to 65535 2019-04-01 12:37:54 -07:00
Chris Lu b4c276263f weed filer: mysql/postgres use ReadCommitted transaction isolation level for atomic rename 2019-04-01 11:03:04 -07:00
Chris Lu 78ac2bef3c go fmt 2019-03-30 23:09:16 -07:00
Chris Lu 97406333a5 support atomic renaming for mysql/postgres filer store 2019-03-30 23:08:29 -07:00
Chris Lu b92122b885 fix test 2019-03-16 09:50:57 -07:00
Chris Lu 36b632ebcb fix tests 2019-03-16 09:50:21 -07:00
Chris Lu 55bab1b456 add context.Context 2019-03-15 17:20:24 -07:00
Chris Lu cece860bfd add context to all filer APIs 2019-03-15 15:55:34 -07:00
Chris Lu 55c85f3a66 adding context 2019-03-15 15:26:09 -07:00
Chris Lu d312c55bbe file path supports windows, avoiding back slashes 
fix https://github.com/chrislusf/seaweedfs/issues/868
 2019-03-04 13:00:08 -08:00
Chris Lu 2442d56671 adjust package name 2019-02-18 18:03:16 -08:00
Chris Lu 77b9af531d adding grpc mutual tls 2019-02-18 12:11:52 -08:00
bingoohuang ab6be025d7 go fmt and fix some typo 2019-01-17 09:17:19 +08:00
Chris Lu 2a75a36b27 set cassandra ttl with entry.TTLSec 2019-01-08 20:26:19 -08:00
byungjo.yoon 58fd2e5a19 Set redis key ttl with entry.TTLSec 2019-01-09 11:48:19 +09:00
Chris Lu be9a7592a1 incrementally calculate visible intervals 2019-01-05 22:23:44 -08:00
Chris Lu 832ac8129a reduce pointer usage 2018-12-31 15:10:14 -08:00
Chris Lu 0388d421d2 caching visible intervals for read 
speeds up 4x in single thread mode
speeds up 30% in 32 threads mode
 2018-12-30 00:51:44 -08:00
Chris Lu 253f190f48 weed mount add symlink support 2018-12-25 22:45:44 -08:00
Chris Lu 3ac54792e1 paginate when filer deleting and FUSE mount renaming 2018-12-16 23:20:08 -08:00
Chris Lu 1bfb96f34d optimization for reading whole chunk with gzip encoding 2018-12-07 01:57:55 -08:00
Chris Lu c28e8a2397 refactoring 2018-12-06 00:44:41 -08:00
Chris Lu 63619f8b8e fail early if two filer stores are defined 2018-12-06 00:32:36 -08:00
Chris Lu 52b24a9902 default "weed server -filer" to same directory as -mdir 2018-12-05 23:24:25 -08:00
Chris Lu b0838db5a8 support hdfs user name and group names 2018-12-04 00:13:40 -08:00
Chris Lu c6b646e92c skip creation for root 2018-12-02 23:16:17 -08:00
Chris Lu f0677c5af1 default root to 0755 2018-12-02 19:59:47 -08:00
Chris Lu db22666a30 add default root directory 2018-12-02 19:42:50 -08:00
Chris Lu 7a6f49cd42 refactor a bit more 
fix https://github.com/chrislusf/seaweedfs/issues/777
 2018-11-29 00:07:54 -08:00
Chris Lu ceca078acb avoid overwriting file or directory 
fix https://github.com/chrislusf/seaweedfs/issues/777
 2018-11-29 00:00:56 -08:00
Chris Lu 86dd933596 go fmt 2018-11-23 00:26:15 -08:00
Chris Lu 99256100a0 fix tests 2018-11-21 16:35:27 -08:00
Chris Lu e5ad2223a1 add missing intervals 
fix https://github.com/chrislusf/seaweedfs/issues/774
 2018-11-21 16:25:13 -08:00
Chris Lu b282e34dc2 async file chunk deletion 2018-11-20 20:56:28 -08:00
Chris Lu b318167cd3 batch delete file ids 2018-11-20 11:35:08 -08:00
Chris Lu ee3fe07acf fix sort bug 2018-11-18 21:59:53 -08:00
Chris Lu 469863f7f0 merge interval now is quite faster with reusing slices 2018-11-18 21:24:58 -08:00
Chris Lu d2924c7748 merge intervals is a bit faster 2018-11-18 20:31:39 -08:00
Chris Lu cb5a10c6a3 adjust the tests 2018-11-18 17:05:36 -08:00
Chris Lu f1db22d48b add benchmark test for merging intervals 2018-11-18 16:49:08 -08:00
Chris Lu 4fcfc9410f cleanup 2018-11-18 11:51:38 -08:00
Chris Lu 9655dc9ca9 simpler logic 2018-11-18 10:07:30 -08:00
Chris Lu a64613172d bootstrap filer replication with weed filer.export -targetStore=notification 2018-11-03 12:43:45 -07:00
Chris Lu 4c97ff3717 support AWS SQS as file change notification message queue 2018-10-31 01:11:19 -07:00
Chris Lu ff66269b62 use grpc to replace http APIs for batch volume id lookup and batch delete 
1. remove batch volume id lookup http API /vol/lookup
2. remove batch delete http API /delete
 2018-10-14 00:12:28 -07:00
Chris Lu 56a5d5af8d ignore duplicated key error if the store has duplicated primary key prevention 
fix https://github.com/chrislusf/seaweedfs/issues/741
 2018-09-30 00:49:52 -07:00
Chris Lu 7bb62b9f21 delay file creation without chunks from dir.Create to file.Flush 2018-09-22 00:11:46 -07:00
Chris Lu db69ce89f0 go fmt 2018-09-21 01:56:43 -07:00
Chris Lu 788acdf527 add WIP filer.replicate 2018-09-17 00:27:56 -07:00
Chris Lu 70e67b7c6d add notification log 2018-09-16 11:20:36 -07:00
Chris Lu d923ba2206 renaming msgqueue to notification 2018-09-16 01:18:30 -07:00
Chris Lu 296acc4a0a go fmt 2018-09-09 16:26:11 -07:00
Chris Lu 164091c269 add s3 multipart upload 2018-09-09 16:25:43 -07:00
Chris Lu b23883b3b6 fix a bug on error reporting 2018-08-22 18:19:26 -07:00
Chris Lu f22d6b87e1 go fmt 2018-08-19 15:42:50 -07:00
Chris Lu 748870b03f add "weed scaffold" command 2018-08-19 15:36:30 -07:00
Chris Lu 95b997fb35 go fmt 2018-08-19 15:18:37 -07:00
Chris Lu f827ada811 merge notification config with filer.toml 2018-08-19 15:17:55 -07:00
Chris Lu ee920d4f83 kafka set the right topic 2018-08-19 01:27:30 -07:00