gitlab-com-seaweedfs/seaweedfs
1
0
Fork 0
mirror of https://github.com/seaweedfs/seaweedfs.git synced 2024-01-19 02:48:24 +00:00
Code Issues Projects Releases Wiki Activity
6264 commits 28 branches 278 tags 135 MiB
Commit graph

4078 commits

Author SHA1 Message Date
Chris Lu 8f7d2d317f readerAt need to use the right offset 
fix https://github.com/chrislusf/seaweedfs/issues/2259
 2021-08-15 11:55:58 -07:00
Chris Lu 0b2a92d371 skip if http.StatusNoContent 2021-08-15 11:27:49 -07:00
Chris Lu ec989b0377 remove println 2021-08-15 02:10:27 -07:00
Chris Lu 9462f5129a shell: add "remote.meta.sync" 2021-08-15 01:53:46 -07:00
Chris Lu 3ada61c875 rename 2021-08-14 21:50:35 -07:00
Chris Lu c34747c79d rename, fix wrong logic. 2021-08-14 21:46:34 -07:00
Chris Lu cb53802752 adjust help message 2021-08-14 15:55:53 -07:00
Chris Lu 889b143fa7 adjust modification detection logic 2021-08-14 15:44:47 -07:00
Chris Lu f365af81c2 parallelize remote content fetching 2021-08-14 15:41:37 -07:00
Chris Lu 53e66980b2 add comments 2021-08-14 15:16:10 -07:00
Chris Lu 9921801e0c Revert "use default or path-specific setting for cache replication level" 
This reverts commit ba6923b223.
 2021-08-14 15:14:26 -07:00
Chris Lu ba6923b223 use default or path-specific setting for cache replication level 2021-08-14 15:14:01 -07:00
Chris Lu 708debca14 remote.cache and uncache: more flexible options to select files to cache or uncache 2021-08-14 15:11:55 -07:00
Chris Lu 08258a819d fix mistake 2021-08-14 05:10:30 -07:00
Chris Lu 4909bd9684 gRpc connection error on filer when no volume left #2243 
fix https://github.com/chrislusf/seaweedfs/issues/2243

grpc do not cache connections only when connection problem happens.
Normal error results should not close the shared grpc connection.
 2021-08-14 05:06:44 -07:00
Chris Lu 8126ab4b5d rename 2021-08-14 05:03:45 -07:00
Chris Lu 0db2517994 go fmt 2021-08-14 02:55:44 -07:00
Chris Lu 7937db52e1 Filer locationPrefix configure does not exec replication #2257 
fix https://github.com/chrislusf/seaweedfs/issues/2257
 2021-08-14 02:54:13 -07:00
Chris Lu 0c66b173a4 fix 2021-08-13 11:31:43 -07:00
Chris Lu e02a317d3d adjust retry logic in case some data is partially written 2021-08-13 11:30:38 -07:00
Chris Lu 2d519c6cb6 adjust the retry logic 2021-08-13 11:13:30 -07:00
Chris Lu b961fcd338 filer: stream read from volume server, reduce memory usage 2021-08-13 11:00:11 -07:00
Chris Lu f4decf02df volume copying: clean up stale volume data files 
fix https://github.com/chrislusf/seaweedfs/issues/2250
 2021-08-13 03:24:21 -07:00
Chris Lu 0f7d4556d8 shell: volume.tier.move makes up changes if volume move failed 2021-08-13 03:09:28 -07:00
Chris Lu 333cdce485 add verbose message 2021-08-13 02:57:14 -07:00
Chris Lu 78e8ddf910 Only when tailing volume, the zero-ed cookie should skip checking. 
This only happens when checkCookie == false and fsync == false.
 2021-08-13 02:09:35 -07:00
Chris Lu a8617c1a39 tail volume: fix zero cookie problem from batch deletion 2021-08-13 01:54:35 -07:00
Chris Lu 5a0f92423e use grpc and jwt 2021-08-12 21:40:33 -07:00
Chris Lu 6238644c35 remove gateway command 2021-08-12 20:52:04 -07:00
Chris Lu d1d1fc772c move some volume lookup operations to grpc 
jwt related lookup will come in next commit
 2021-08-12 20:33:00 -07:00
Chris Lu 5571f4f70a master: add master.follower to handle read file id lookup requests 2021-08-12 18:10:59 -07:00
Chris Lu 5469019852 adjust data type 2021-08-12 17:54:34 -07:00
Chris Lu bfac55e6c0 avoid integer overflow 
fix https://github.com/chrislusf/seaweedfs/issues/2254
 2021-08-11 22:22:49 -07:00
Chris Lu 5516fa8e80
Merge pull request #2251 from kmlebedev/rertyUpdateIamIdentity 
Retry save and update IAM identity
 2021-08-11 09:21:03 -07:00
Konstantin Lebedev f0afd35eec Retry save and update IAM identity 
https://github.com/chrislusf/seaweedfs/issues/2242
 2021-08-11 19:29:04 +05:00
byunghwa.yun 775dfbae85 Synchronize number of open files 2021-08-11 23:14:56 +09:00
Konstantin Lebedev ec09966fd3 Retry save and update IAM identity 
https://github.com/chrislusf/seaweedfs/issues/2242
 2021-08-11 17:53:48 +05:00
Chris Lu f2cd753bf9 fix avoid lock error 
fix https://github.com/chrislusf/seaweedfs/issues/2247
 2021-08-10 14:34:13 -07:00
Chris Lu 01336d71eb minor 2021-08-10 13:04:33 -07:00
Chris Lu 1154e23e2d add logs for volume creation 2021-08-10 13:04:25 -07:00
Chris Lu 057ef429ac format 2021-08-10 12:33:29 -07:00
Chris Lu b63b042afc dedup keeps the largest replica 2021-08-10 12:30:41 -07:00
Chris Lu 0526db12e2 do not treat read only volumes differently 2021-08-10 11:37:12 -07:00
Chris Lu e50a5b8e28 minor: print disk type 2021-08-10 11:10:09 -07:00
Chris Lu db6275a0c8 print out balance ratio 2021-08-10 10:43:42 -07:00
Chris Lu 9d85569c55 ensure using local quorum consistency 2021-08-10 05:10:57 -07:00
Chris Lu 85832d02c0 wait for goroutines 2021-08-10 04:13:12 -07:00
Bl1tz23 1c94b3d013 merge master, resolve conflicts 2021-08-10 13:45:24 +03:00
Bl1tz23 e6e57db530 Add liveness\readiness probe for s3 api handler on /status path 2021-08-10 13:42:46 +03:00
Chris Lu 8ff6c9a0c6 output format 2021-08-10 03:25:18 -07:00
Chris Lu 48f448ee09 parallelize tier move 2021-08-10 03:08:29 -07:00
Chris Lu 69a6da7969 avoid fail on tail error 2021-08-10 02:50:28 -07:00
Chris Lu 18228f3044 fix help message 2021-08-10 02:48:41 -07:00
Chris Lu 69655ba8e5 mount: cache on reading remote storage 2021-08-09 22:11:57 -07:00
Chris Lu a7012d9729 fix 2021-08-09 16:03:03 -07:00
Chris Lu 02f728cb54 fix bug with remote.uncache 2021-08-09 15:24:21 -07:00
Chris Lu 8d3e275735 remote: filer cache remote content on read 2021-08-09 15:16:45 -07:00
Chris Lu 9096f6f4f7 cache: set upper limit of chunk size 2021-08-09 15:08:53 -07:00
Chris Lu 402315f117 go fmt 2021-08-09 14:37:34 -07:00
Chris Lu a6be2520c9 fix 2021-08-09 14:37:25 -07:00
Chris Lu 713c035a6e shell: remote.cache remote.uncache 2021-08-09 14:35:18 -07:00
Chris Lu 8cfd487608 2.62 2021-08-08 23:33:12 -07:00
Chris Lu 734c980040 volume: support concurrent download data size limit 2021-08-08 23:25:16 -07:00
Chris Lu 96ce85f5ae rename 2021-08-08 22:33:31 -07:00
Chris Lu c5f38c365d go fmt 2021-08-08 22:30:36 -07:00
Chris Lu df85f7a1eb adjust help message 2021-08-08 22:30:12 -07:00
Chris Lu c0b12da4ef shell: add filer.remote.unmount 2021-08-08 22:26:37 -07:00
Chris Lu 882a93dacd fix tests 2021-08-08 17:56:26 -07:00
Chris Lu 7412ccdf88 write back remote entry to local entry after uploading to remote 2021-08-08 17:55:03 -07:00
Chris Lu dcf614a8c3 skip if the remote entry update is because of internal managerial operations 2021-08-08 15:58:10 -07:00
Chris Lu eed26af266 Merge branch 'master' into add_remote_storage 2021-08-08 15:48:04 -07:00
Chris Lu 4370a4db63 use int64 for volume count in case of negative overflow 2021-08-08 15:19:39 -07:00
Chris Lu 26c222f596 shell: volume.tier.move avoid moving all volumes to one destination 2021-08-08 15:12:39 -07:00
Chris Lu 13e45e1605 filer.remote.sync can work now 2021-08-08 01:21:42 -07:00
Chris Lu 8f5170c138 remove imports 2021-08-07 16:20:17 -07:00
Chris Lu 46b0cb8c86 Merge branch 'master' into add_remote_storage 2021-08-07 15:41:27 -07:00
Chris Lu de730b079d ChunkStreamReader implenents io.ReaderAt 2021-08-07 15:41:07 -07:00
Chris Lu 59732a0529 refactoring 2021-08-07 15:35:27 -07:00
Chris Lu ecb234f75a refactor 2021-08-07 14:46:23 -07:00
Chris Lu 270770d7d7 refactor 2021-08-07 14:18:53 -07:00
Chris Lu 679f800caa Merge branch 'master' into add_remote_storage 2021-08-06 20:27:12 -07:00
Chris Lu 0c0f77e2ae skip not found error on deletion 2021-08-06 19:35:47 -07:00
Chris Lu 8828f485c0 print volume deletion error 2021-08-06 19:30:22 -07:00
Chris Lu d2b23f3d38
Merge pull request #2238 from combineads/change_default_mode 
Change default permissions
 2021-08-05 21:28:06 -07:00
Chris Lu e23ca3845b
Merge pull request #2237 from combineads/add_mtime 
Add crtime and mtime
 2021-08-05 21:27:13 -07:00
Chris Lu 286e5dd375 Merge branch 'master' into add_remote_storage 2021-08-05 21:07:04 -07:00
Chris Lu 1e22166939 adjust error message 2021-08-05 21:06:55 -07:00
byunghwa.yun f3dc909b21 Change default permissions 2021-08-06 12:24:35 +09:00
byunghwa.yun e84fad9acf Add crtime and mtime 2021-08-06 12:13:37 +09:00
Chris Lu 1a5d29520c add tests 2021-08-05 14:49:24 -07:00
Chris Lu 6b743dbbf9 refactor client subscribe metadata 2021-08-04 16:25:46 -07:00
Chris Lu b9ecf1e3a8 refacotring 2021-08-04 14:56:13 -07:00
Chris Lu d84c311699 refactoring 2021-08-04 12:30:18 -07:00
Chris Lu f6a9ad8001 fix tests 2021-08-04 00:31:06 -07:00
Chris Lu 42969c9c62 Merge branch 'master' into add_remote_storage 2021-08-03 21:20:05 -07:00
Chris Lu 8cf0c515bf shell: volume.fix.replication retries even when there is no error #2235 
fix https://github.com/chrislusf/seaweedfs/issues/2235
 2021-08-03 09:32:55 -07:00
Chris Lu d2ddf1dbdb typo 2021-08-03 09:32:31 -07:00
Chris Lu 3afbf04007 add TLS grpc support for filer meta clients 2021-08-03 01:30:35 -07:00
Chris Lu fe60c6ef9a minor changes 2021-08-02 11:49:40 -07:00
byunghwa.yun bdc7730fdb Add autocomplete 2021-08-03 02:56:52 +09:00
Chris Lu 2ae9705442 adjust text 2021-08-01 22:55:19 -07:00
Chris Lu 3739717092 Revert "adds a test" 
This reverts commit f690643b47.
 2021-08-01 22:54:45 -07:00
Chris Lu f690643b47 adds a test 2021-08-01 22:53:50 -07:00
Chris Lu 6de786185d volume.balance: balance read only volumes first 2021-08-01 22:32:50 -07:00
Chris Lu 3bb640b786 add -force option following #2228 2021-08-01 20:03:05 -07:00
byunghwa.yun 9e839cb5cc Add force option in volume.deleteEmpty command 2021-08-02 11:37:20 +09:00
Chris Lu 56ee1d5ef1 2.61 2021-08-01 15:50:19 -07:00
Chris Lu 81b255df8b Merge branch 'master' into add_remote_storage 2021-08-01 15:44:21 -07:00
Chris Lu 58bc3ecf47 add default quietFor value 2021-08-01 15:36:06 -07:00
Chris Lu 2ca1839d77 shell: add volume.deleteEmpty command 2021-08-01 15:33:45 -07:00
Chris Lu 89933c46d2 s3: skip hidden directories in /buckets folder 2021-08-01 12:28:08 -07:00
Chris Lu 1c7e404abe remove buckets folder option 
the related code still works for old deployments
 2021-08-01 12:23:16 -07:00
Chris Lu cb1dbd3135 refactor 2021-08-01 11:53:46 -07:00
Chris Lu 1161c70dab Merge branch 'master' into add_remote_storage 2021-08-01 01:10:13 -07:00
Chris Lu 9cc84a910f volume: deletion can skip volume size checking 
fix https://github.com/chrislusf/seaweedfs/issues/2225
 2021-08-01 00:32:51 -07:00
Chris Lu 767edd3c08 rename 2021-07-31 23:52:09 -07:00
Chris Lu 9df7d16791 read <- remote_storage 2021-07-31 22:39:38 -07:00
Chris Lu f5a69a0e44 Merge branch 'master' into add_remote_storage 2021-07-31 09:39:48 -07:00
Chris Lu 6ba65c3382 customizable debug port 2021-07-31 09:18:41 -07:00
byunghwa.yun 5d4438a72f Fix typo 2021-07-31 22:23:52 +09:00
Chris Lu 1ff8285d82 debug from any server 2021-07-31 02:13:21 -07:00
Chris Lu 0d6f45cb46 add debug option to generate full goroutine stack dump 2021-07-31 02:00:01 -07:00
Chris Lu a3290faf17 shell command to calculate size / number of volumes in a collection 
fix https://github.com/chrislusf/seaweedfs/issues/2224
 2021-07-30 15:49:46 -07:00
Chris Lu c6f992b2a3 remove dead code 2021-07-30 15:18:01 -07:00
Chris Lu 899963ac20 remote storage location changed to struct 2021-07-29 02:08:55 -07:00
Chris Lu c090d6bb25 add ReadRemote(), add read remote setup when filer starts 2021-07-28 22:43:12 -07:00
Chris Lu 4deac06da0 log fix 2021-07-28 22:25:04 -07:00
divanikus 5c6270a93a shell: ability to use wildcards for collections, all collections if ommited 2021-07-27 23:53:01 +03:00
Chris Lu 035b0bae29 refactor 2021-07-27 03:32:24 -07:00
Chris Lu 1752eeb538 remote.mount saves the mapping 2021-07-27 03:26:35 -07:00
Chris Lu 4b94b03d90 directory to remote storage mapping 2021-07-27 01:16:28 -07:00
Chris Lu 99b599aa8a remote.mount 2021-07-26 22:53:44 -07:00
Chris Lu 5dede5d38d 2.60 2021-07-25 22:09:09 -07:00
Chris Lu 35f70c51b0 refactor 2021-07-25 03:37:37 -07:00
Chris Lu 72eec84167 shell: add fs.mkdir 2021-07-25 02:37:39 -07:00
byunghwa.yun 2b28a818f1 Fix mysql sql for batch delete 2021-07-25 11:06:14 +09:00
Chris Lu 10464f47d0
Merge pull request #2217 from combineads/add_webp_format 2021-07-24 09:42:41 -07:00
byunghwa.yun 7374b5b473 Add webp extension for resizing 2021-07-24 14:32:31 +09:00
byunghwa.yun 2595f269d1 Add the webp image type 2021-07-24 14:26:40 +09:00
Chris Lu ac28611817 snowflake sequencer need an unique id 
fix https://github.com/chrislusf/seaweedfs/issues/2213
 2021-07-23 20:54:03 -07:00
Chris Lu 5c14da0f1e filer.copy: fail early if assign request failed 
fix https://github.com/chrislusf/seaweedfs/issues/2216
 2021-07-23 20:01:43 -07:00
Chris Lu 4cc2165061 weed -h subcommand list is not sorted, makes discovery difficult 
fix https://github.com/chrislusf/seaweedfs/issues/2214
 2021-07-23 18:44:53 -07:00
Chris Lu 10fc478557 scaffold -config= should output to stdout 
fix https://github.com/chrislusf/seaweedfs/issues/2212
 2021-07-23 18:41:25 -07:00
Chris Lu 092932af8f shell: faster bucket delete 
avoid waiting for large object metadata
 2021-07-23 11:25:46 -07:00
Chris Lu bdb632fa62
Merge pull request #2211 from qieqieplus/filer-bloom-filter 
add bloom filter for filer leveldb/rocksdb
 2021-07-22 23:35:54 -07:00
qieqieplus 2b46df13f8 add bloom filter for filer leveldb/rocksdb 2021-07-23 14:05:59 -06:00
Chris Lu 60f5c0a2f5 fix security alert on github.com/dgrijalva/jwt-go 
resolve https://github.com/chrislusf/seaweedfs/security/dependabot/go.mod/github.com%2Fdgrijalva%2Fjwt-go/open
 2021-07-22 10:22:09 -07:00
Chris Lu 182288f860 filer: fix mysql, postgres batch delete error 2021-07-22 08:23:20 -07:00
Chris Lu 84d91f143f mount: hide /etc folder also from mount 2021-07-22 00:40:16 -07:00
Chris Lu 7359193e97 go fmt 2021-07-21 14:38:12 -07:00