gitlab-com-seaweedfs/seaweedfs
1
0
Fork 0
mirror of https://github.com/seaweedfs/seaweedfs.git synced 2024-01-19 02:48:24 +00:00
Code Issues Projects Releases Wiki Activity
5639 commits 28 branches 278 tags 135 MiB
Commit graph

3613 commits

Author SHA1 Message Date
Chris Lu c131764c34 ensure error is sent back in channel 
fix https://github.com/chrislusf/seaweedfs/issues/1966

avoid shared readErr variable
 2021-04-05 19:40:12 -07:00
Chris Lu 0f64f5b9c8 mount: add readOnly option 
fix https://github.com/chrislusf/seaweedfs/issues/1961
 2021-04-04 21:40:58 -07:00
Chris Lu 6eee200c13 2.37 2021-04-04 18:45:48 -07:00
Chris Lu fbb82a5c9c skip limiting if limit is zero 2021-04-04 18:38:33 -07:00
Chris Lu 8251d1140e refactor 2021-04-04 18:38:33 -07:00
Chris Lu 5dca98fe71 fix typo 2021-04-03 11:37:13 -07:00
Chris Lu 5e64f65632 ensure tables are created 
fix https://github.com/chrislusf/seaweedfs/issues/1957
 2021-04-02 12:02:26 -07:00
Chris Lu 319743d330 go fmt 2021-04-02 02:22:59 -07:00
Chris Lu 8e404a1433 go fmt 2021-04-02 02:22:26 -07:00
Chris Lu bdf2ddddfd revert to same implementation as before 
This reverts commit 7e8edc3c4a.
 2021-04-02 02:21:38 -07:00
Chris Lu 67e019d54b add missing changes 2021-04-02 01:11:44 -07:00
Chris Lu 7e8edc3c4a refactoring 2021-04-02 01:10:24 -07:00
Chris Lu cc0df36a9e smaller file chunks are faster 2021-04-01 02:21:59 -07:00
Chris Lu cefe66f159 dedicated upload processor 
avoid thundering effect of overloading volume servers
 2021-04-01 02:21:40 -07:00
Chris Lu 1f984d2645 refactor buffer pool 2021-04-01 02:20:00 -07:00
Chris Lu b5880334fc refactor 2021-03-30 21:07:34 -07:00
Chris Lu e79e2ddeed remove unused MoveEvents 2021-03-30 20:42:44 -07:00
Chris Lu e06807a55b adjust logs 2021-03-30 20:36:06 -07:00
Chris Lu ac875976c0 filer, volume: add concurrent upload size limit to avoid OOM 
add some back pressure when writes are slow
 2021-03-30 02:10:53 -07:00
Chris Lu a1e18a1384 server add memory profiling 2021-03-30 02:10:53 -07:00
LazyDBA247-Anyvision 9385e2224d fix postgres2 upsertQuery syntax also 2021-03-30 10:51:52 +03:00
Chris Lu a5662c9e5e adjust ordering 2021-03-30 00:26:57 -07:00
Chris Lu bec74c3e6a
Merge pull request #1948 from LazyDBA247-Anyvision/master 
add enableUpsert=true
 2021-03-30 00:25:01 -07:00
LazyDBA247-Anyvision 9f1cab179c use a compatible pg syntax for upsert 2021-03-30 10:21:27 +03:00
Chris Lu a95929e53c reduce default concurrentWriters to 32 2021-03-30 00:17:52 -07:00
LazyDBA247-Anyvision 7f44d953b5 fix GetBool 2021-03-30 01:36:02 +03:00
LazyDBA247-Anyvision 4c51e6a660 add enableUpsert=true 
and rename config to upsertQuery
 2021-03-30 00:32:03 +03:00
Chris Lu b61e9f6a1a adjust ordering 2021-03-29 00:49:50 -07:00
Chris Lu 1ebdbad222
Merge pull request #1946 from LazyDBA247-Anyvision/master 
Adding Customisation - insertQuery for postgres/mysql
 2021-03-29 00:42:48 -07:00
Konstantin Lebedev 8a95f9c10c iam GetUser 2021-03-29 12:01:44 +05:00
LazyDBA247-Anyvision 96c62bd34d scaffold - insertQuery 
Example for upsert query for mysql & postgres variants
 2021-03-29 10:01:12 +03:00
LazyDBA247-Anyvision 4a02389eb0 Adding custom insertQuery support for postgres/2 mysql/2 2021-03-29 09:58:13 +03:00
Chris Lu 6b7aa9633f 2.36 2021-03-28 19:09:06 -07:00
Chris Lu bd7471d877 refactor 2021-03-25 12:05:59 -07:00
李海 06be5dc6c3 log snowflake sequencer nodeid's hex when start, it'll be in part of new assigned fid 2021-03-25 18:59:40 +08:00
李海 69b2dab9c6 add a snowflake sequencer as more robust fid generator, but less compressable than small auto-inc id 2021-03-25 18:49:26 +08:00
Konstantin Lebedev 40938d6a47 SaveInsideFiler S3 Configuration 2021-03-25 12:34:11 +05:00
Konstantin Lebedev 9f26f2815c SaveAs S3 Configuration 2021-03-25 12:34:11 +05:00
Konstantin Lebedev 82b0463fac handler PutUserPolicy 
https://docs.aws.amazon.com/IAM/latest/APIReference/API_PutUserPolicy.html
 2021-03-25 12:34:11 +05:00
Konstantin Lebedev d7719d0542 base handlers 2021-03-25 12:34:11 +05:00
Konstantin Lebedev 03c7953254 init Iam Api Server 2021-03-25 12:34:11 +05:00
Chris Lu 82dfe06066
Merge pull request #1933 from kmlebedev/missingKeysThreshold 
add missingKeysThreshold for fsck
 2021-03-24 10:56:25 -07:00
Konstantin Lebedev 020a5d40c3 avoid counter is null 2021-03-24 22:51:22 +05:00
Konstantin Lebedev df6cf0a2fa nonRepairThreshold 2021-03-24 22:07:13 +05:00
Konstantin Lebedev 9ce82a9cd2 GOMAXPROCS use defaults to the value of runtime.NumCPU 
https://github.com/chrislusf/seaweedfs/issues/1942
 2021-03-24 18:45:58 +05:00
Konstantin Lebedev 60972f1c97 rename option repairThreshold 2021-03-24 13:24:49 +05:00
Chris Lu a801332b0d filer: return 409 if file conflicts with a directory 
fix https://github.com/chrislusf/seaweedfs/issues/1938
 2021-03-24 01:03:11 -07:00
Chris Lu c42b95c596 more help message 
fix https://github.com/chrislusf/seaweedfs/issues/1937
 2021-03-23 17:27:57 -07:00
Konstantin Lebedev 40a9e88c07 rename var 2021-03-23 21:42:35 +05:00
Konstantin Lebedev d21a5bf5d0 add missingKeysThreshold for fsck 2021-03-23 15:04:07 +05:00
Chris Lu 85cff10787 mysql: ensure name column is case sensitive 
fix https://github.com/chrislusf/seaweedfs/issues/1931
 2021-03-23 00:46:50 -07:00
Chris Lu 4abb511db3 make a local copy of the in memory cached data 2021-03-22 22:33:07 -07:00
Chris Lu 3cbc40fa48 avoid creating multiple reader 2021-03-22 22:32:47 -07:00
Chris Lu 2f7c7afdec refactor: remove unused parameter 2021-03-22 22:13:19 -07:00
Chris Lu d48dd0c738 use []byte directly instead of bytes.Buffer 2021-03-22 22:12:57 -07:00
Chris Lu 8eb14967ef shell: volume.check.disk adds optional slow mode 
fix https://github.com/chrislusf/seaweedfs/issues/1930
 2021-03-22 21:01:43 -07:00
Chris Lu 288369cfc7 mount: release resources when Forget() is called 
address https://github.com/chrislusf/seaweedfs/issues/1929
 2021-03-22 16:22:19 -07:00
Chris Lu 5d931eff27 avoid possible nil 
fix https://github.com/chrislusf/seaweedfs/issues/1928

The nil was because of `dn.Parent().UnlinkChildNode(dn.Id())` in topo.UnRegisterDataNode() function, when the dn leaves the cluster.
 2021-03-22 13:24:07 -07:00
Chris Lu f315eb2bb8 fix build error 2021-03-22 00:12:53 -07:00
Chris Lu 1dd5bc134c 2.35 2021-03-22 00:05:09 -07:00
Chris Lu b465095db1 shell: add volume.check.disk to fix inconsistency for replicated volumes 
fix https://github.com/chrislusf/seaweedfs/issues/1923
 2021-03-22 00:03:16 -07:00
Chris Lu df461402cc ensure entry attribute is not nill 
fix https://github.com/chrislusf/seaweedfs/issues/1926
 2021-03-21 22:27:30 -07:00
Chris Lu 102a951377 refactor, split into 2 files 2021-03-21 13:05:53 -07:00
Chris Lu f6e2566ee5 better logs for cookie mismatch 2021-03-20 12:02:57 -07:00
Chris Lu 27e24162f2 minor 2021-03-20 06:37:03 -07:00
Chris Lu 41cc6e661b fs.mv ensures there are 2 arguments 2021-03-20 06:34:13 -07:00
Chris Lu b2c1c209a5 refactor 2021-03-20 06:34:13 -07:00
Chris Lu e9b1853d63 go fmt 2021-03-19 01:31:56 -07:00
Chris Lu b1a86cf808 s3: copy object to itself 
fix https://github.com/chrislusf/seaweedfs/issues/1922
 2021-03-19 01:31:49 -07:00
Konstantin Lebedev bf94149920 add line 2021-03-17 23:41:34 +05:00
Konstantin Lebedev c5705e7a35 dir + slash key does not exist 
https://github.com/chrislusf/seaweedfs/issues/1917
 2021-03-17 23:40:42 +05:00
Konstantin Lebedev eb54993a4e Merge branch 'upstreamMaster' into check_chunkviews_mr 
# Conflicts:
#	weed/filer/filechunk_manifest.go
#	weed/filer/stream.go
#	weed/replication/repl_util/replication_util.go
#	weed/util/fasthttp_util.go
 2021-03-16 15:29:49 +05:00
Chris Lu 9672f9e1b2 2.34 2021-03-16 03:01:15 -07:00
Chris Lu 69694a17be reverting 7d57664c2d 2021-03-16 02:59:26 -07:00
Chris Lu c00dd5e62e report error if entry not found 2021-03-16 02:55:49 -07:00
Konstantin Lebedev 06da02739d CheckAllChunkViews() for HEAD requests only 2021-03-16 14:15:17 +05:00
Chris Lu 91a3ac9731 2.33 2021-03-16 00:36:06 -07:00
Chris Lu 4b1ed227d1 revert fasthttp changes 
related to https://github.com/chrislusf/seaweedfs/issues/1907
 2021-03-16 00:33:14 -07:00
Chris Lu 10164d0386
Merge pull request #1905 from wuh-fnst/s3ListCounterErr 
s3: count correction for the number of files in the dir
 2021-03-15 20:54:58 -07:00
Chris Lu 22a07a12d2 [mount] [regression] filer.path option results in empty mount in 2.32 
fix https://github.com/chrislusf/seaweedfs/issues/1906
 2021-03-15 20:46:25 -07:00
wuh-fnst 050a7f9599 s3: count correction for the number of files in the dir 2021-03-16 09:22:03 +08:00
Konstantin Lebedev 3a3699867b Status PartialContent for Content-Range response 2021-03-15 23:30:22 +05:00
Konstantin Lebedev 90510e3137 If WriteHeader is not called explicitly, the first call to Write 
will trigger an implicit WriteHeader(http.StatusOK).
WriteHeader: Only one header may be written. Go does not currently!
 2021-03-15 21:22:59 +05:00
Konstantin Lebedev 46b9f5cff4 add debug logging 2021-03-15 19:33:33 +05:00
Konstantin Lebedev 7194a5e7bf avoid http error: superfluous response.WriteHeader 
https://github.com/chrislusf/seaweedfs/issues/1838
 2021-03-15 18:52:59 +05:00
Chris Lu 9f00f95bfb 2.32 2021-03-14 21:32:00 -07:00
Chris Lu a4cfffc264 shell: fix moving volume, volume server evacuate 
fix https://github.com/chrislusf/seaweedfs/issues/1534
 2021-03-14 21:29:55 -07:00
Chris Lu ca100568f7 fix parsing 2021-03-14 20:50:14 -07:00
Chris Lu 352ba23f83 revert previous change 
revert 29e62aba00
 2021-03-14 20:49:56 -07:00
Chris Lu 29e62aba00 possible fix for volume balance 
address https://github.com/chrislusf/seaweedfs/issues/1534
 2021-03-14 20:23:19 -07:00
Chris Lu 35d939a1d2 go fmt 2021-03-14 13:21:02 -07:00
Chris Lu 2d4c2db81d filer: leveldb, rocksdb auto create store directory 
fix https://github.com/chrislusf/seaweedfs/issues/1901
 2021-03-14 13:20:14 -07:00
Chris Lu 6d3a96eb56 filer: mysql2, postgres2 trigger actions on bucket creation and deletion 
fix https://github.com/chrislusf/seaweedfs/issues/1877
 2021-03-13 22:07:39 -08:00
Chris Lu cb423312a4 prevent nil volume message 2021-03-13 11:05:29 -08:00
Chris Lu 972327f966 prevent nil volume nm 2021-03-13 11:04:51 -08:00
Chris Lu 2a68ddb963 default to empty host to bind to tcp4 and tcp6 2021-03-12 15:32:51 -08:00
Chris Lu 5fbcaaf37a fullpath() test is not needed 2021-03-12 09:45:25 -08:00
Chris Lu 7d57664c2d mount: internals switch to filer.Entry instead of protobuf 2021-03-12 00:36:38 -08:00
wuh-fnst 4c1d945e46 make List correctly judge whether it is the last file 2021-03-12 11:22:26 +08:00
Chris Lu cca66c7fbe print out the url 2021-03-11 18:39:45 -08:00
Chris Lu 7f887c78d2 fix 2021-03-11 18:18:16 -08:00
Chris Lu fcc52d1e4e shell: add s3.clean.uploads to clean up stale multipart uploads 
fix https://github.com/chrislusf/seaweedfs/issues/1855
 2021-03-11 18:05:24 -08:00
Chris Lu 40dc5ac904 mount: add a bit retry when connecting to filer during startup 2021-03-11 16:36:43 -08:00
Chris Lu aba47fd9e8 reduce repeated calls to dir.FullPath() 2021-03-11 14:08:20 -08:00
Chris Lu d084334ffd
Merge pull request #1883 from kmlebedev/passDelNonEmptyFolder 
s3 delete-objects pass OK for fail to delete non-empty folder
 2021-03-11 11:01:44 -08:00
Chris Lu 0e35836cc1
Merge pull request #1890 from kmlebedev/ZeroDataSize 
error if read chunk zero data size
 2021-03-11 10:51:48 -08:00
Konstantin Lebedev 58cdcc6d6e error if urls not found 2021-03-11 23:34:36 +05:00
Chris Lu 79280e1513 escape special characters 
fix https://github.com/chrislusf/seaweedfs/issues/1884
 2021-03-11 09:49:40 -08:00
Konstantin Lebedev 4db21012c1 error if read chunk zero data size 2021-03-11 22:38:59 +05:00
wuheng 828fbf3fb7 s3: "isLast" returns true when the file does not exist 2021-03-11 15:20:50 +08:00
Chris Lu 5d446673a6 fix error printing 2021-03-10 17:13:04 -08:00
Chris Lu ca546f47e1 s3: escape object key if containing special characters 
fix https://github.com/chrislusf/seaweedfs/issues/1884
 2021-03-10 13:19:28 -08:00
Chris Lu 105fcd7bfc
Merge pull request #1872 from kmlebedev/grpc_auth 
TLS allowed CommonNames
 2021-03-10 11:17:49 -08:00
Konstantin Lebedev bb3af2d70b no directoriesWithDeletion 2021-03-10 19:52:41 +05:00
Konstantin Lebedev 15b018da34 s3 delete-objects pass OK for fail to delete non-empty folder 2021-03-10 19:41:35 +05:00
Konstantin Lebedev 348e21a08c add comments 2021-03-10 14:42:39 +05:00
Konstantin Lebedev 831953c55c allowed wildcard domain 2021-03-10 14:02:13 +05:00
Konstantin Lebedev 4bf93d6e63 comma-separated 2021-03-10 12:43:13 +05:00
Konstantin Lebedev 0e02f7e258 comma-separated SSL certificate common names 2021-03-10 12:42:44 +05:00
Chris Lu 737bde5ab7 refactoring, remove duplicated code 2021-03-09 23:23:01 -08:00
Chris Lu 5ba4b479f8 properly lock file.entry object 
fix https://github.com/chrislusf/seaweedfs/issues/1882
 2021-03-09 23:08:38 -08:00
Chris Lu 5be83dd4a9 add comments 2021-03-09 14:13:48 -08:00
Chris Lu 9d8ca513d9 refactor 2021-03-09 13:21:26 -08:00
Chris Lu f5b5e4da2d 2.31 2021-03-09 12:52:16 -08:00
Chris Lu 828f6e9f4d volume: auto add missing vif files 
fix https://github.com/chrislusf/seaweedfs/issues/1878
 2021-03-09 12:09:32 -08:00
Chris Lu 387c6f4218 Revert "delete the folder object in multi_object_delete" 
This reverts commit 2e89c8c9ae.
 2021-03-09 10:07:27 -08:00
qieqieplus 5b16820924 fix: collection stats won't update if all volumes expired at same iteration 2021-03-09 18:49:45 +08:00
Konstantin Lebedev 190fada1ef TLS allowed commonNames 2021-03-08 21:39:44 +05:00
Konstantin Lebedev c6d3735605 permitCommonNames 
https://github.com/chrislusf/seaweedfs/issues/1841
https://jbrandhorst.com/post/grpc-auth/
 2021-03-08 13:16:17 +05:00
Chris Lu d888e9a90f 2.30 2021-03-07 15:01:19 -08:00
Chris Lu 60686a0bb7 make tcp optional 2021-03-07 14:45:36 -08:00
Chris Lu 3b2a9c98ef fix nil 2021-03-07 11:45:35 -08:00
Chris Lu df72dc206d stats collect chunk upload retry count 2021-03-07 11:26:15 -08:00
Chris Lu 726edab054 avoid nil when closing an index 
fix https://github.com/chrislusf/seaweedfs/issues/1870
 2021-03-07 11:03:09 -08:00
Chris Lu bdfed16d42 avoid nil exception 
fix https://github.com/chrislusf/seaweedfs/issues/1869
 2021-03-07 01:49:06 -08:00
Chris Lu 1bd880dcdb adds tcp writes benchmark 2021-03-06 14:26:27 -08:00
Chris Lu 9d402ebe9f refactoring 2021-03-06 14:26:27 -08:00
Chris Lu e9eddfb8a2 remove wrong writes 2021-03-06 14:26:27 -08:00
Chris Lu 8363be8548 trim out trailing return character 2021-03-06 14:26:27 -08:00
Chris Lu 38fc200e56 CRCWriter consistent with CRC 2021-03-06 14:26:27 -08:00
Chris Lu 1444e9d275 migrated multi host connection pool from godropbox package 
removing unneeded dependencies, which involved etcd versions.
 2021-03-06 14:26:27 -08:00
Patrick Schmidt 7413d59750 Fix EC shard count logic 
This fixes the calculation of the amount of EC shards a node holds.
Previously a global counter was increased, but also used inside the
loop to apply disk usage deltas. This led to wrong absolute numbers.
The fix is to apply only deltas of single EC shards per iteration.
 2021-03-05 12:50:58 +01:00
Chris Lu 400de380f4 volume server: support tcp direct put/get/delete 2021-03-05 02:29:38 -08:00
Chris Lu 2e89c8c9ae delete the folder object in multi_object_delete 
fix https://github.com/chrislusf/seaweedfs/issues/1857
 2021-03-04 14:03:40 -08:00
Chris Lu c5cb346a08 fix compilation error 2021-03-04 12:36:20 -08:00
Chris Lu f5cc96f289 skip checking master peers if not starting master 2021-03-04 12:31:26 -08:00
Chris Lu 40ff30b83f adjust help message 2021-03-03 02:12:46 -08:00
Chris Lu c0842fe99f add filer.meta.backup command 2021-03-03 02:02:29 -08:00
Chris Lu 0c5c51eb98 adjust help message 2021-03-02 23:07:39 -08:00
Chris Lu bd727b7b53 go fmt 2021-03-02 20:59:56 -08:00