gitlab-com-seaweedfs/seaweedfs
1
0
Fork 0
mirror of https://github.com/seaweedfs/seaweedfs.git synced 2024-01-19 02:48:24 +00:00
Code Issues Projects Releases Wiki Activity
4315 commits 28 branches 278 tags 135 MiB
Commit graph

2672 commits

Author SHA1 Message Date
Chris Lu b61d33f251 expose only store wrapper to meta changes 2020-09-22 16:27:36 -07:00
Chris Lu 0adbb56cc1 rename 2020-09-22 16:24:13 -07:00
Chris Lu 3e52329cee Revert "Merge pull request #1479 from LIBA-S/fix_oversized" 
This reverts commit bd11f0b3e4, reversing
changes made to ec5b9f1e91.
 2020-09-22 15:05:37 -07:00
Chris Lu 4a1fe4b8e2 add error logs 2020-09-22 09:16:10 -07:00
LIBA-S 8573ac82c6 Fix: remove the oversized state after compaction 2020-09-22 21:48:39 +08:00
James Hartig 658fc2e5b6 Allow option to enable volume pprof on server 2020-09-21 22:43:10 -04:00
James Hartig 91e4eca1e9 Fix deadlock with KeepConnected and SendHeartbeat 
There's the potential where we're writing to a clientConn and it goes away
and we're stuck keeping a read lock on clientChansLock. This causes
KeepConnected to not be able to remove the client since it requires a write
lock on clientChansLock. This ends up backing up SendHeartbeat because it
can't get a read lock.
 2020-09-21 22:41:38 -04:00
Chris Lu 9cdbfc1a49 refactor 2020-09-21 17:34:38 -07:00
Chris Lu 63373a9f9f filer: add file extended properties 2020-09-21 11:08:34 -07:00
Chris Lu b9c1f3e9de s3: fixes for list multipart upload 2020-09-21 10:51:24 -07:00
Chris Lu 289e62a305 master: better locking of in memory volume data 
related to https://github.com/chrislusf/seaweedfs/issues/1436#issuecomment-695880135
 2020-09-20 23:07:55 -07:00
Chris Lu 9a3b564508 adjust printout 2020-09-20 16:01:56 -07:00
Chris Lu 62563a895a refactoring 2020-09-20 16:00:01 -07:00
Chris Lu c2faab23b6 refactor 2020-09-20 15:40:49 -07:00
Chris Lu d013d09a9b adjust logging 2020-09-20 15:38:59 -07:00
Chris Lu f498c71199 shell: move volume operations to use flag parsing arguments 2020-09-20 09:27:34 -07:00
Chris Lu fcbc520373 fix print out 2020-09-20 08:47:30 -07:00
Chris Lu 56094541c8 shell: adjust command option for collection.delete 2020-09-20 08:46:16 -07:00
Chris Lu 9cac9c4001 2.00 2020-09-20 00:47:04 -07:00
Chris Lu dc4d2145ff open up 2020-09-19 20:23:12 -07:00
Chris Lu 29abe980df s3: add support for PostPolicy 
fix https://github.com/chrislusf/seaweedfs/issues/1426
 2020-09-19 20:14:19 -07:00
Chris Lu 41d508edfd go fmt 2020-09-19 14:10:26 -07:00
Chris Lu 5b40a2690a refactoring 2020-09-19 14:09:58 -07:00
Chris Lu 2c21eb1971 volume: get metrics configuration from master 
fix https://github.com/chrislusf/seaweedfs/issues/1354
 2020-09-19 00:03:00 -07:00
Chris Lu 2cbd1cf121 fix compilation 2020-09-18 00:15:54 -07:00
Chris Lu 23e9ede068 s3: collect metrics 2020-09-18 00:09:04 -07:00
Chris Lu 852e5f7cbc filer: fix mongodb insert 
fix https://github.com/chrislusf/seaweedfs/issues/1471
 2020-09-17 21:50:52 -07:00
Chris Lu 6544e60bea s3 add metrics 
empty for now
 2020-09-17 06:56:15 -07:00
Chris Lu cb427d48fa filer report metrics configuration 2020-09-17 06:46:51 -07:00
Chris Lu e4e0234998 refactoring 2020-09-17 06:43:54 -07:00
Chris Lu feca07bf96 Filer: ip bind 
fix https://github.com/chrislusf/seaweedfs/issues/1470
 2020-09-16 10:26:08 -07:00
Chris Lu e861a6a3ab simplify metrics configuration logic 2020-09-16 01:39:30 -07:00
Chris Lu be54eeb364 passing value by checkWithMaster() instead 2020-09-16 01:33:45 -07:00
Chris Lu 4d21de63ee go fmt 2020-09-16 01:27:24 -07:00
Chris Lu 913a16268d volume: load configuration from master at the start 
fix https://github.com/chrislusf/seaweedfs/issues/1469
 2020-09-16 01:27:05 -07:00
Chris Lu c9202c4b3d add storage backend to GetMasterConfigurationResponse 2020-09-16 01:12:22 -07:00
Chris Lu 8a0710cb73 handle more than 2GB files 
related to https://github.com/chrislusf/seaweedfs/issues/1468
 2020-09-16 00:41:50 -07:00
Chris Lu 854007bc98 minor 2020-09-16 00:37:57 -07:00
Chris Lu 10f9081526 filer: adjust meta data events to received timestamp 
if a client is already connected and start from t0. A message recieved at t+1 but with timestamp t-1 may not be processed by the client.

This commit changes to the event received time, so the replication can be ordered.
 2020-09-15 01:18:33 -07:00
Chris Lu 684a875876 update help message 2020-09-15 00:40:38 -07:00
Chris Lu 2d3a904a82 shell: volumeServer.evacuate adds option to skip non moveable volumes 2020-09-15 00:33:49 -07:00
Chris Lu a595916342 shell: add volumeServer.evacuate command 2020-09-14 23:47:11 -07:00
Chris Lu 103fafe00b weed export: print out [start,stop) content range 2020-09-14 22:57:23 -07:00
Chris Lu ada996fe55 filer: support createing empty folder 
fix https://github.com/chrislusf/seaweedfs/issues/1161
 2020-09-14 13:07:46 -07:00
Chris Lu 200fe5c83e go fmt 2020-09-13 21:26:30 -07:00
Chris Lu 5d6753fb98 shell: add volumeServer.leave command 2020-09-13 21:25:51 -07:00
Chris Lu 1af95c5b76 refactoring 2020-09-13 12:41:26 -07:00
Chris Lu f8fea19669 1.99 2020-09-12 13:46:33 -07:00
Chris Lu b0c7de186d filer: fix postgres prefixed directory listing problem 
fix https://github.com/chrislusf/seaweedfs/issues/1465
 2020-09-12 13:37:03 -07:00
Chris Lu 1a7afe7e6a adjust default value 
related to https://github.com/chrislusf/seaweedfs/issues/1453
 2020-09-12 12:46:42 -07:00
Chris Lu f2723c1bc8 do not idx file format 
revert c9ab8d05fa
 2020-09-12 12:42:36 -07:00
Chris Lu ba984a4e29 1.98 2020-09-12 04:13:02 -07:00
Chris Lu 446e476a11 go fmt 2020-09-12 04:08:03 -07:00
Chris Lu ea26a98753 volume: validate volume correctness if last entry is a deletion 2020-09-12 04:07:04 -07:00
Chris Lu d15682b4a1 shell: volume.balance plan by ratio of fullness 2020-09-12 04:06:26 -07:00
Chris Lu c0ee78d2fa adjust make file 2020-09-12 04:05:42 -07:00
Chris Lu cd9b89ba55 reduce default wait time to 10s before shutting down 2020-09-12 04:05:33 -07:00
Chris Lu e2c741f76f adjust replica placement after move 2020-09-12 01:01:19 -07:00
Chris Lu 2a0925590c filer: etcd store avoid read with nil option 
fix https://github.com/chrislusf/seaweedfs/issues/1463
 2020-09-11 15:29:45 -07:00
Chris Lu 3984c3962f add comment 2020-09-11 15:07:19 -07:00
Chris Lu baa6bdf4d4 s3: listMultipartUploads fix output format 2020-09-11 15:04:01 -07:00
Chris Lu 3eda8d6dfc s3: ListParts output xml format 
fix https://github.com/chrislusf/seaweedfs/issues/1461
 2020-09-11 14:53:50 -07:00
Chris Lu ab201c2798 1.97 2020-09-11 13:47:50 -07:00
Chris Lu 9d4bdfcfdf fix volume integrity checking 2020-09-11 11:34:10 -07:00
Chris Lu 1b8094ef75 weed export: export deleted files 2020-09-11 02:05:14 -07:00
Chris Lu 12a8f5294d test for multi dc replication 2020-09-11 01:21:17 -07:00
Chris Lu e60b2117c3 shell: volume balance follows replica placement 2020-09-11 00:29:25 -07:00
Chris Lu 89a62e8007 refactoring 2020-09-10 23:05:00 -07:00
Chris Lu 19537c9d21 1.96 2020-09-10 19:52:07 -07:00
Chris Lu eaf9fdde99 avoid sharing context over separate goroutine 2020-09-10 19:51:03 -07:00
Chris Lu cc5fe6f5ee handle special characters in html link 2020-09-10 19:46:02 -07:00
Chris Lu 401ccf1509 fix test 2020-09-10 15:13:23 -07:00
Chris Lu c9ab8d05fa fixes for reading deleted fid 2020-09-10 14:42:52 -07:00
ruitao.liu 5b0676049a change elastic initialize process similar as others. 2020-09-10 23:35:20 +08:00
Chris Lu 660d7c0edd 1.95 2020-09-10 01:32:05 -07:00
Chris Lu 0d9b858cfb
Merge pull request #1455 from taozix/master 
add more basic elastic options.
 2020-09-10 01:31:17 -07:00
ruitao.liu 9be4e97625 change logs print format. 2020-09-10 16:30:15 +08:00
ruitao.liu 719dc43af1 modify elastic urls from string to array. 2020-09-10 16:24:09 +08:00
ruitao.liu 72f9d7f047 use util to generate md5. 2020-09-10 16:11:18 +08:00
ruitao.liu 6a5b38c0d4 fix elastic kv ops. 2020-09-10 15:59:16 +08:00
ruitao.liu 3f7fbfddca add more basic elastic options. 2020-09-10 14:22:07 +08:00
Chris Lu b183ae54b5 master: changing 301 to 308 on redirect to volume servers 
fix https://github.com/chrislusf/seaweedfs/issues/1454
 2020-09-09 22:36:10 -07:00
Chris Lu 47b3f932e4 watch: adjust output format 2020-09-09 22:34:48 -07:00
Chris Lu daf0a449f7 properly cancel context for streaming grpc 2020-09-09 12:07:15 -07:00
Chris Lu 7f69acd1f2 sync pprof 2020-09-09 11:33:52 -07:00
Chris Lu 387ab6796f filer: cross cluster synchronization 2020-09-09 11:21:23 -07:00
Chris Lu 4fc0bd1a81 return http response directly 2020-09-09 03:53:09 -07:00
Chris Lu 5e13bc878c adjust log level 2020-09-09 03:49:26 -07:00
Chris Lu d1b816212f return nil if no need to do anything 2020-09-07 16:03:05 -07:00
Chris Lu 64a621bcc8 shell: volume.fix.replication also purge over replicated volumes 2020-09-07 16:00:10 -07:00
Chris Lu c18ea21f7a 1.94 2020-09-07 13:13:53 -07:00
Chris Lu 18b98fdb72 fix test 2020-09-07 12:57:38 -07:00
Chris Lu 44b3f2efc8 filer: Elastic Search return correct kv error 
fix https://github.com/chrislusf/seaweedfs/issues/1452
 2020-09-07 12:44:51 -07:00
Chris Lu d80538a187 refactoring 2020-09-07 12:35:02 -07:00
Chris Lu 1a09bc43d1 refactor 2020-09-07 11:31:33 -07:00
Chris Lu 2b643f477d typo 2020-09-06 12:47:55 -07:00
Chris Lu 05034aade5 printout over replicated locations 2020-09-06 12:44:02 -07:00
Chris Lu 432fe3d926 1.93 2020-09-06 12:12:09 -07:00
Chris Lu d8af6f62df adjust logging 2020-09-06 12:10:40 -07:00
Chris Lu 07f32feed0 print sync progress 2020-09-06 00:50:51 -07:00
Chris Lu 0af6252b2d fix compilation 2020-09-06 00:50:38 -07:00
Chris Lu ac494ff5e8 Update meta_aggregator.go 2020-09-06 00:29:16 -07:00
Chris Lu c9f8f25ba5 read peer filer from start 2020-09-06 00:12:41 -07:00
Chris Lu 30dc365cbd add self filer 2020-09-06 00:11:46 -07:00
Chris Lu 8acd714600 apply meta changes only if store is different 2020-09-05 23:25:03 -07:00
Chris Lu d741ed66db check whether shares the same filer store 2020-09-05 23:02:52 -07:00
Chris Lu bba90ff3c8 read filer signature 2020-09-05 22:52:15 -07:00
Chris Lu 71b0e256c7 filer: setOrLoadFilerStoreSignature 2020-09-05 14:08:59 -07:00
ruitao.liu 450cf07505 skip the index that for kv usage. 2020-09-04 21:49:03 +08:00
ruitao.liu 1384ff9a2f 1.split kv in one file. 
2.disable query for kv in es index.
 2020-09-04 17:34:26 +08:00
ruitao.liu 83080b5e03 ES backended filer support kv ops. 2020-09-04 15:40:13 +08:00
Chris Lu 46f65a84a6 filer: elastic7 adjust default value 2020-09-03 20:12:38 -07:00
Chris Lu f815fce81b
Merge pull request #1446 from taozix/master 
new filer store option to es7.
 2020-09-03 20:10:23 -07:00
Chris Lu 44b4ebf0ff filer: Redis cleanly delete directory 
fix https://github.com/chrislusf/seaweedfs/issues/1448
 2020-09-03 19:17:39 -07:00
Chris Lu b8f32bcab9 filer: compress stored metadata 2020-09-03 11:00:20 -07:00
Chris Lu f76a2b2c8a printout meta data size 2020-09-03 09:51:21 -07:00
ruitao.liu 798280e980 change filer2 to filer. 2020-09-03 17:05:26 +08:00
ruitao.liu a93d27d1e8 new filer option to es v7. 2020-09-03 16:34:58 +08:00
Chris Lu 1d56ea24ef fix 2020-09-03 00:08:37 -07:00
Chris Lu 7e1aad0b54 mount: map uid/gid between local and filer 2020-09-03 00:07:22 -07:00
Chris Lu 68e878adb5 fix formatting 2020-09-02 21:42:12 -07:00
Chris Lu 0d99a5da3c c*: support kv operations 2020-09-02 18:39:24 -07:00
Chris Lu 645a4af3db mongodb: support kv operations 2020-09-02 17:19:14 -07:00
Chris Lu ecaa30c408 better error message 2020-09-02 17:17:44 -07:00
Chris Lu 7c770b727c Update abstract_sql_store_kv.go 2020-09-02 10:13:48 -07:00
Chris Lu 05a6e2dc7f sql kv upsert 2020-09-02 10:12:44 -07:00
Chris Lu ed62f52452 fix sql insert with duplicated primary key 2020-09-02 10:09:49 -07:00
Chris Lu 9ea290aa12 fix error message 2020-09-01 23:21:41 -07:00
Chris Lu 06a1188267 mysql/postgres: support kv operations 2020-09-01 22:48:36 -07:00
Chris Lu aa40295f06 leveldb: report not found entry 2020-09-01 22:48:23 -07:00
Chris Lu 87d7312bf6 mysql/postgres: properly report entry not found 2020-09-01 22:47:57 -07:00
Chris Lu 72b0a5f1d1 mysql or postgres: log find error 2020-09-01 22:25:17 -07:00
Chris Lu 2815bbe6c0 go fmt 2020-09-01 21:59:26 -07:00
Chris Lu 37234bf3f8 filer store adds kv support 
can compile now, need to implement those unimplemented
 2020-09-01 21:59:00 -07:00
James Hartig 8e54e34576 volume: Don't unmount before deleting volume in copy 
If we unmount first and then delete, the delete fails because the volume
was unmounted. Delete ends up doing the same thing as the unmount anyways.
 2020-09-01 22:00:07 -04:00
Chris Lu 2b14ae5819 fix tests 2020-09-01 01:33:43 -07:00
Chris Lu d91ec535b3 fix tests 2020-09-01 01:29:13 -07:00
Chris Lu eb7929a971 rename filer2 to filer 2020-09-01 00:21:19 -07:00
Chris Lu 38e06d783d volume: check disk space before compaction 
fix https://github.com/chrislusf/seaweedfs/issues/1440
 2020-08-31 18:10:53 -07:00
Chris Lu dbb10e0f0b testing with more than 30days 2020-08-31 17:59:29 -07:00
Chris Lu dd176cdb8b avoid tiny step pagination 2020-08-31 11:28:03 -07:00
Chris Lu 97733c3dff typo 2020-08-31 10:49:17 -07:00
Chris Lu c8398bdb09 adjust metrics 2020-08-31 10:41:05 -07:00
Chris Lu 8a1c8e41b3 simplify if else logic 2020-08-31 10:39:24 -07:00
Chris Lu f0c89cfacd go fmt 2020-08-31 10:23:31 -07:00
Chris Lu edb9d65e05
Merge pull request #1431 from kmlebedev/wip-prefix-search 
Wip prefix search
 2020-08-31 10:22:14 -07:00
Konstantin Lebedev 9a195bebfd accurate limit 2020-08-31 22:13:13 +05:00
Konstantin Lebedev 18b98f9747 not convert err to string 2020-08-31 21:55:18 +05:00
Konstantin Lebedev 22f32e75c4 rename 2020-08-31 21:52:05 +05:00
Konstantin Lebedev a8de332beb fix 2020-08-31 21:45:44 +05:00