Commit graph

5581 commits

Author SHA1 Message Date
Evgeny Kuzhelev 47c72e6f35 remove all (currently existing) collection volume metrics 2022-07-27 16:31:49 +05:00
Evgeny Kuzhelev 709c83716c delete disk_size metrics when collection deleted 2022-07-27 15:02:41 +05:00
chrislu f401b996eb mount: fix symlink size reporting 2022-07-26 23:12:25 -07:00
Konstantin Lebedev c88ea31f62 fix RUnlock of unlocked RWMutex 2022-07-26 12:57:07 +05:00
Konstantin Lebedev da9d3e8f6c refactor 2022-07-26 11:56:45 +05:00
Konstantin Lebedev 046c3d5ad4 fix logic else brake 2022-07-26 11:47:11 +05:00
chrislu 475185fb72 3.18 2022-07-25 22:41:38 -07:00
chrislu ec0edb1ac4 filer: fix wrong logic during read 2022-07-25 22:40:00 -07:00
guosj cc7a4b0a6e correct comment 2022-07-25 11:46:41 +08:00
chrislu 33cff61992 fix glog test 2022-07-24 19:13:54 -07:00
chrislu dbe4849ffd 3.17 2022-07-24 18:46:50 -07:00
chrislu b98c0b4bf4 disable symlink translation for now 2022-07-24 18:46:32 -07:00
chrislu 2dc5d4adac disable symlink translation for now 2022-07-24 18:24:34 -07:00
chrislu 3d2bbf96d2 adjust glog format 2022-07-24 17:29:08 -07:00
chrislu 2807fe08fc adjust glog format 2022-07-24 16:44:58 -07:00
chrislu 2b64916d3e set status to fuse.OK 2022-07-24 15:30:55 -07:00
chrislu 7386cf6a06 mount: resolve symlink 2022-07-24 15:11:24 -07:00
chrislu 40089b6753 maintain node lookup counter 2022-07-24 13:43:11 -07:00
chrislu d24db396cc fix removing path from inode2path 2022-07-23 18:20:29 -07:00
chrislu 41eeb4deef do not add new inode during link 2022-07-23 13:37:07 -07:00
chrislu 02c2d81cde fix removePathFromInode2Path 2022-07-23 13:21:42 -07:00
chrislu 4d5dc55766 mount: better support hard links 2022-07-23 12:57:00 -07:00
chrislu ebda0689ab use linux default max name length 2022-07-23 10:51:06 -07:00
chrislu 64f3d6fb6e metadata subscription uses client epoch 2022-07-23 10:50:28 -07:00
chrislu 2c8818351f mount: comment out stuck fuse inode notify during renaming 2022-07-22 18:17:34 -07:00
chrislu a5c94acd4f mount: check possible nil entry case 2022-07-22 15:33:37 -07:00
Chris Lu 0716092b39
Merge pull request #3350 from shichanglin5/optimize_masterclient_vidmap
Solve the problem that `LookupFileId` lookup urls is empty due to lea…
2022-07-22 02:57:50 -07:00
LHHDZ 994a2dec78 fix vid_map_test tests 2022-07-22 17:54:57 +08:00
LHHDZ 58c02d6429 Solve the problem that LookupFileId lookup urls is empty due to leader switching
The vidMap structure is modified to a linked list structure (the length is limited to 5). When the vidMap is reset, the current vidMap is added to the new vidMap as a cache node. When the query locations is empty, the cache node is searched to avoid problems when the master switches leaders.
2022-07-22 17:22:38 +08:00
ningfd 58f2dd6740 change topology mapping definition, improve spelling 2022-07-22 17:18:56 +08:00
ningfd 6f882eb354 optimiz: master ui will render data in order 2022-07-22 14:37:49 +08:00
chrislu 7a6c559ab4 fix Change replication via volume.configure.replication by collection
fix https://github.com/chrislusf/seaweedfs/issues/3346
2022-07-21 22:01:05 -07:00
chrislu fc8241fb5e leveldb3: add instant dropping bucket 2022-07-21 18:48:51 -07:00
chrislu c93f7ffa44 explicit bucket aware declaration 2022-07-21 18:23:53 -07:00
Konstantin Lebedev 3c42814b58 avoid deadlock 2022-07-21 17:15:10 +05:00
Konstantin Lebedev 93ca87b7cb use safe onPeerUpdateDoneCns 2022-07-21 15:51:14 +05:00
chrislu 6147b61b0a mount: fix concurrent map read and map write
fix https://github.com/chrislusf/seaweedfs/issues/3344
2022-07-21 01:38:26 -07:00
Chris Lu 9ec0d1caaa
Merge pull request #3337 from guol-fnst/loading_volume 2022-07-20 09:06:06 -07:00
Konstantin Lebedev 7875470e74 onPeerUpdateGoroutineCount use int32 2022-07-20 18:40:50 +05:00
Konstantin Lebedev c5189c343b remove ticker
update the topology before each file
2022-07-20 18:40:50 +05:00
Konstantin Lebedev e2d991d8d0 ticker.Stop 2022-07-20 18:40:50 +05:00
Konstantin Lebedev de4fcc0e2c sync update topologyInfo 2022-07-20 18:40:50 +05:00
Konstantin Lebedev fa88dff7ce update otherNodes 2022-07-20 18:40:50 +05:00
Konstantin Lebedev 72dca31cfa fix update topologyInfo 2022-07-20 18:40:50 +05:00
Konstantin Lebedev 884ffbafee clouse background update 2022-07-20 18:40:50 +05:00
Konstantin Lebedev 39eaf426f8 fix TestVolumeServerEvacuate 2022-07-20 18:40:50 +05:00
Konstantin Lebedev b5e5f6f55a update topologyInfo 2022-07-20 18:40:50 +05:00
Konstantin Lebedev 867269cdcf help rack 2022-07-20 18:40:50 +05:00
Konstantin Lebedev 6f764e1014 volume server evacuate from rack 2022-07-20 18:40:50 +05:00
Konstantin Lebedev ba0e3ce5fa volume server evacuate to target server 2022-07-20 18:40:50 +05:00
Konstantin Lebedev b6471ecd75 err msg with duplicated local subscription detected move to log level 1
https://github.com/chrislusf/seaweedfs/issues/3320
2022-07-20 18:40:50 +05:00
Konstantin Lebedev 6c390851e7 fix design 2022-07-20 18:08:12 +05:00
Tuan Vuong 312e13416b filter duplicated action 2022-07-20 16:43:50 +07:00
guol-fnst ac694f0c8f rename parameter and reuse functions
rename milestone to  watermark
2022-07-20 17:00:40 +08:00
Chris Lu 4820b594a7
Merge pull request #3298 from kmlebedev/evacuateTargetServer
volume server evacuate to target server
2022-07-19 13:56:49 -07:00
Konstantin Lebedev d3f7c09c03 remove ticker
update the topology before each file
2022-07-20 00:54:23 +05:00
Konstantin Lebedev 11e393dbe7 err msg with duplicated local subscription detected move to log level 1
https://github.com/chrislusf/seaweedfs/issues/3320
2022-07-20 00:45:13 +05:00
Konstantin Lebedev f6a966b4fc add waiting log message 2022-07-20 00:31:57 +05:00
Konstantin Lebedev 6cfbfb0849 check for ping before deleting raft server
https://github.com/chrislusf/seaweedfs/issues/3083
2022-07-20 00:04:12 +05:00
chrislu 2ae3f812f8 minor 2022-07-19 11:43:31 -07:00
Chris Lu 9667588af0
Merge pull request #3318 from kmlebedev/issues/3310
Use fallback if urls are not found
2022-07-19 11:38:43 -07:00
Chris Lu f0d560060f
Merge pull request #3336 from kmlebedev/raftSrvRemove
raftServerRemoval typo and logs
2022-07-19 11:32:17 -07:00
guol-fnst 91285bb51d remove uncessary code 2022-07-19 20:00:44 +08:00
guol-fnst b9256e0b34 optimiz 2022-07-19 16:58:53 +08:00
Konstantin Lebedev f419d5643a fix typo
add remove logs
2022-07-19 11:50:52 +05:00
guol-fnst bec9f79659 use snapshot 2022-07-19 11:14:41 +08:00
Konstantin Lebedev d422e7769c ticker.Stop 2022-07-18 16:38:19 +05:00
Konstantin Lebedev 73a0dea16b sync update topologyInfo 2022-07-18 16:27:02 +05:00
chrislu 56ec89625a 3.16 2022-07-18 00:53:05 -07:00
Konstantin Lebedev 2b4112e462 update otherNodes 2022-07-18 11:32:28 +05:00
Konstantin Lebedev 3c2774ec3d fix update topologyInfo 2022-07-18 01:46:31 +05:00
Konstantin Lebedev 7b1497ee63 Use BackoffSchedule for getLookupFileId 2022-07-15 16:05:35 +05:00
Konstantin Lebedev 01996bccf8 Use fallback if urls are not found 2022-07-15 15:29:15 +05:00
Konstantin Lebedev 4d5144e50d clouse background update 2022-07-15 13:51:08 +05:00
chrislu 1db012485f mount: fix bug during busy writes
fix https://github.com/chrislusf/seaweedfs/issues/3315
2022-07-15 01:03:17 -07:00
chrislu aca20cd9f4 more logging related to filer metadata subscription 2022-07-15 00:42:13 -07:00
Chris Lu abeb7f65f8
Merge pull request #3314 from shichanglin5/fix_ETag
File upload should succeed should return `ETag` instead of `Etag`
2022-07-14 15:35:41 -07:00
chrislu fa61074513 add clientId logging 2022-07-14 12:27:34 -07:00
chrislu fbd8f868a1 filer may have trouble to re-connect
clientId is used twice: one for local metadata subscription, one for combined metadata subscription.
2022-07-14 12:15:31 -07:00
LHHDZ c08f6d0ff7 File upload should succeed should return ETag instead of Etag
at `/go-1.18.1/src/net/textproto/header.go:39`, header is canonicalized by `textproto.CanonicalMIMEHeaderKey`
2022-07-14 19:45:01 +08:00
Chris Lu 957c9a2eb2
Merge pull request #3309 from guol-fnst/loading_volume
optimiz concurrency
2022-07-13 23:12:24 -07:00
guol-fnst 300b383cdf use 10 or numCPU workers if env is not found 2022-07-14 14:06:46 +08:00
guol-fnst 308a48c0c2 optimiz concurrency
user can customize number of workers via env "GOMAXPROCS"
2022-07-14 09:57:25 +08:00
Chris Lu 1a130125de
Merge pull request #3308 from guo-sj/check_duplicate_ak 2022-07-13 09:23:37 -07:00
guosj 354b7bdff0 replace errors.New(fmt.Sprintf(...)) with fmt.Errorf() 2022-07-13 22:49:03 +08:00
chrislu 289402a741 writer pattern: similar changes to reader pattern 2022-07-13 02:30:44 -07:00
guosj ab1b9697e6 supplement check duplicate accesskey 2022-07-13 17:28:20 +08:00
chrislu 6e90f7bdd0 detect sequential mode with a counter 2022-07-13 02:20:03 -07:00
chrislu 578c8aea0e typo 2022-07-13 02:18:31 -07:00
chrislu 1451b389a4 Avoid fatal error: sync: Unlock of unlocked RWMutex
fix https://github.com/chrislusf/seaweedfs/issues/3306
2022-07-13 00:58:15 -07:00
石昌林 392da4e038 Make the prompt information clearer 2022-07-13 11:39:45 +08:00
石昌林 ad5e8f68ec Check whether there is a duplicate accessKey when modifying iam 2022-07-13 11:36:11 +08:00
Chris Lu fba5219dab
Merge pull request #3303 from guol-fnst/loading_volume
speeding up loading volumes
2022-07-12 03:01:24 -07:00
Konstantin Lebedev 6622240df7 fix TestVolumeServerEvacuate 2022-07-12 14:56:34 +05:00
chrislu 860c207c94 typo 2022-07-12 02:56:19 -07:00
Konstantin Lebedev 8372721a62 update topologyInfo 2022-07-12 13:47:21 +05:00
guol-fnst 26313060a3 speeding up loading volumes 2022-07-12 16:19:33 +08:00
Konstantin Lebedev ee95d23a22 help rack 2022-07-12 11:56:58 +05:00
Konstantin Lebedev 087fa1347f volume server evacuate from rack 2022-07-12 11:33:08 +05:00
Chris Lu de97ba94c3
Merge pull request #3302 from guo-sj/fix_put_user_policy 2022-07-11 21:15:37 -07:00
guosj b3076909d1 refactor code 2022-07-12 11:25:36 +08:00
Konstantin Lebedev 4236c36599 volume server evacuate to target server 2022-07-11 16:58:15 +05:00
Chris Lu 7929a50327
Merge pull request #3296 from guo-sj/fix_put_user_policy
refine PutUserPolicy
2022-07-11 01:49:25 -07:00
guosj 7dbf19d09a refine PutUserPolicy 2022-07-11 16:38:41 +08:00
chrislu 37578929d4 fix compilation 2022-07-11 01:30:00 -07:00
chrislu 93ccc6e05f 3.15 2022-07-11 00:50:07 -07:00
chrislu f87952a411 remove locks 2022-07-11 00:25:20 -07:00
guosj 5469efa336 close SkipCheckParentDirectory flag 2022-07-11 09:44:48 +08:00
chrislu 3d0defa663 fix compilation 2022-07-08 19:16:58 -07:00
chrislu 6f00c7f061 comments 2022-07-08 19:07:31 -07:00
chrislu d685b94102 mount: rename also invalidate source inode 2022-07-08 19:07:03 -07:00
chrislu 48382676d2 fix filtering by volume id 2022-07-08 10:29:24 -07:00
liubaojiang 4e83e92156 add subscriber clientId if it is the first time 2022-07-08 20:27:25 +08:00
guosj 5b3192344d fix iam CreatePolicy error 2022-07-08 17:42:21 +08:00
chrislu e2ecf137a2 fix resetting fh.reader 2022-07-08 01:04:15 -07:00
chrislu 185332b91f Merge branch 'master' of https://github.com/chrislusf/seaweedfs 2022-07-08 00:29:41 -07:00
chrislu 28add5a534 mount: fix racing conditions
prevent wrong reading when the SingleChunkCacher is started, but not finished yet
2022-07-08 00:29:39 -07:00
duanhongyi 1ceab96aba filer tikv support tls 2022-07-08 14:23:06 +08:00
chrislu a85ed3fe8f minor 2022-07-07 22:27:24 -07:00
chrislu 2ea18cdcc8 remove dead code 2022-07-07 22:26:03 -07:00
chrislu b7de5c6c43 shell: remove unused filer parameter from shell.toml 2022-07-07 15:01:23 -07:00
chrislu f2f0482dd3 mount: random read also try to use the local cache first 2022-07-07 11:50:28 -07:00
chrislu 05c53820b9 mount: file handler release memory 2022-07-07 11:49:40 -07:00
chrislu 69ef6459c3 S3 ListObjectsV2 fix pagination under bucket root
fix https://github.com/chrislusf/seaweedfs/issues/3166
2022-07-06 02:28:34 -07:00
Konstantin Lebedev 785223e587 rabbitpubsub enable durable 2022-07-06 10:05:29 +05:00
chrislu 57975b93ff adjust used size reporting 2022-07-04 13:47:42 -07:00
chrislu a79f5bd0d5 adjust used size reporting 2022-07-04 13:46:32 -07:00
chrislu 3c79c77056 3.14 2022-07-04 13:15:49 -07:00
guol-fnst 5bc895673e add getting bucket quota 2022-07-04 15:36:44 +08:00
guol-fnst dd4664e610 fix usage 2022-07-04 11:31:39 +08:00
chrislu 141f662734 edge case checking when volume server does not have capacity to balance
fix https://github.com/chrislusf/seaweedfs/issues/3257
2022-07-01 12:16:18 -07:00
chrislu bee3a7c798 fix tests 2022-07-01 12:09:42 -07:00
Chris Lu ddb0f6bc74
Merge pull request #3258 from kmlebedev/issues_3255
volumeServer.evacuate fix select empty node
2022-07-01 02:36:30 -07:00
Konstantin Lebedev 3d5f1cbc5e fix pagination 2022-07-01 14:17:49 +05:00
chrislu 06a8b174b5 also remove Sync() for idx file 2022-06-30 13:50:53 -07:00
chrislu 52580743b9 stuck with file.Sync()
starting to get weed/storage () - (master) > go test ./...
I0630 12:19:54 65819 needle_map_metric_test.go:26] FileCount expected 10000 actual 10000
I0630 12:19:54 65819 needle_map_metric_test.go:27] DeletedSize expected 1648 actual 1648
I0630 12:19:54 65819 needle_map_metric_test.go:28] ContentSize expected 10000 actual 10000
I0630 12:19:54 65819 needle_map_metric_test.go:29] DeletedCount expected 1648 actual 1959
I0630 12:19:54 65819 needle_map_metric_test.go:30] MaxFileKey expected 10000 actual 10000
I0630 12:19:54 65819 volume_loading.go:136] loading index /var/folders/jv/0rlhdck17jzgt7b3hcnq63mc0000gp/T/TestCompaction909350722/001/1.idx to memory
I0630 12:20:36 65819 volume_vacuum.go:98] Committing volume 1 vacuuming...
panic: test timed out after 10m0s

goroutine 61 [running]:
testing.(*M).startAlarm.func1()
	/usr/local/go/src/testing/testing.go:2029 +0x8e
created by time.goFunc
	/usr/local/go/src/time/sleep.go:176 +0x32

goroutine 1 [chan receive, 10 minutes]:
testing.(*T).Run(0xc0000cd520, {0x1c09bde?, 0x5ab798ea5c102?}, 0x1c67bd8)
	/usr/local/go/src/testing/testing.go:1487 +0x37a
testing.runTests.func1(0xc0000cd520?)
	/usr/local/go/src/testing/testing.go:1839 +0x6e
testing.tRunner(0xc0000cd520, 0xc00039fcd8)
	/usr/local/go/src/testing/testing.go:1439 +0x102
testing.runTests(0xc0000dadc0?, {0x25b4460, 0x6, 0x6}, {0x2855108?, 0x40?, 0x25c4b80?})
	/usr/local/go/src/testing/testing.go:1837 +0x457
testing.(*M).Run(0xc0000dadc0)
	/usr/local/go/src/testing/testing.go:1719 +0x5d9
main.main()
	_testmain.go:57 +0x1aa

goroutine 21 [chan receive]:
github.com/chrislusf/seaweedfs/weed/glog.(*loggingT).flushDaemon(0x0?)
	/Users/chrislu/go/src/github.com/chrislusf/seaweedfs/weed/glog/glog.go:883 +0x6a
created by github.com/chrislusf/seaweedfs/weed/glog.init.0
	/Users/chrislu/go/src/github.com/chrislusf/seaweedfs/weed/glog/glog.go:410 +0x1bf

goroutine 55 [syscall]:
syscall.syscall(0x44ac0f96?, 0xc00007ad80?, 0xc00098b480?, 0x10588df?)
	/usr/local/go/src/runtime/sys_darwin.go:22 +0x4e
syscall.fcntl(0x100000001?, 0xc00098b4b0?, 0x1069a7c?)
	/usr/local/go/src/syscall/zsyscall_darwin_amd64.go:319 +0x30
internal/poll.(*FD).Fsync.func1(...)
	/usr/local/go/src/internal/poll/fd_fsync_darwin.go:18
internal/poll.ignoringEINTR(...)
	/usr/local/go/src/internal/poll/fd_posix.go:74
internal/poll.(*FD).Fsync(0xc00038f1e0?)
	/usr/local/go/src/internal/poll/fd_fsync_darwin.go:17 +0xfc
os.(*File).Sync(0xc00012a030)
	/usr/local/go/src/os/file_posix.go:168 +0x4e
github.com/chrislusf/seaweedfs/weed/storage/backend.(*DiskFile).Sync(...)
	/Users/chrislu/go/src/github.com/chrislusf/seaweedfs/weed/storage/backend/disk_file.go:84
github.com/chrislusf/seaweedfs/weed/storage.(*Volume).makeupDiff(0xc0000bb440, {0xc000633a40, 0x52}, {0xc000633aa0, 0x52}, {0xc000633b00, 0x52}, {0xc000633b60, 0x52})
	/Users/chrislu/go/src/github.com/chrislusf/seaweedfs/weed/storage/volume_vacuum.go:295 +0x12fa
github.com/chrislusf/seaweedfs/weed/storage.(*Volume).CommitCompact(0xc0000bb440)
	/Users/chrislu/go/src/github.com/chrislusf/seaweedfs/weed/storage/volume_vacuum.go:119 +0x3e8
github.com/chrislusf/seaweedfs/weed/storage.TestCompaction(0xc00025a000)
	/Users/chrislu/go/src/github.com/chrislusf/seaweedfs/weed/storage/volume_vacuum_test.go:89 +0x305
testing.tRunner(0xc00025a000, 0x1c67bd8)
	/usr/local/go/src/testing/testing.go:1439 +0x102
created by testing.(*T).Run
	/usr/local/go/src/testing/testing.go:1486 +0x35f

goroutine 56 [chan receive, 10 minutes]:
github.com/chrislusf/seaweedfs/weed/storage.(*Volume).startWorker.func1()
	/Users/chrislu/go/src/github.com/chrislusf/seaweedfs/weed/storage/volume_write.go:244 +0x41a
created by github.com/chrislusf/seaweedfs/weed/storage.(*Volume).startWorker
	/Users/chrislu/go/src/github.com/chrislusf/seaweedfs/weed/storage/volume_write.go:234 +0x56
FAIL	github.com/chrislusf/seaweedfs/weed/storage	600.194s
?   	github.com/chrislusf/seaweedfs/weed/storage/backend	[no test files]
ok  	github.com/chrislusf/seaweedfs/weed/storage/backend/memory_map	(cached)
?   	github.com/chrislusf/seaweedfs/weed/storage/backend/s3_backend	[no test files]
ok  	github.com/chrislusf/seaweedfs/weed/storage/erasure_coding	(cached)
?   	github.com/chrislusf/seaweedfs/weed/storage/idx	[no test files]
ok  	github.com/chrislusf/seaweedfs/weed/storage/needle	(cached)
ok  	github.com/chrislusf/seaweedfs/weed/storage/needle_map	(cached)
ok  	github.com/chrislusf/seaweedfs/weed/storage/super_block	(cached)
?   	github.com/chrislusf/seaweedfs/weed/storage/types	[no test files]
?   	github.com/chrislusf/seaweedfs/weed/storage/volume_info	[no test files]
FAIL
2022-06-30 12:32:55 -07:00
Konstantin Lebedev 5ed8165161 fix logic
add option targetServer
https://github.com/chrislusf/seaweedfs/issues/3255
2022-06-30 17:58:24 +05:00
石昌林 200a862701 fixed volume xx not found caused by missing VolumeLocation events
When the requested master node is the leader, `VolumeLocation` or `ClusterNodeUpdate` may be returned here. If it is `VolumeLocation`, the update will be performed while resetting the vidMap, otherwise the event will be lost
2022-06-30 13:41:56 +08:00
Chris Lu 4ae9884a6e
Merge pull request #3254 from shichanglin5/fix_subscribe_meta
[Critical] Avoid synchronizing from scratch due to err
2022-06-29 12:36:28 -07:00
Chris Lu b98556cff5
Merge pull request #3253 from kmlebedev/constMultipartUploadsFolder
use const multipart uploads folder
2022-06-29 12:09:03 -07:00
石昌林 2b5b37231b Make sure that onChangeEvent is called successfully before updating lastTsNs 2022-06-29 23:57:07 +08:00
石昌林 e17e6ed542 Avoid synchronizing from scratch due to err 2022-06-29 23:31:04 +08:00
Konstantin Lebedev bcbdc4cb37 use const multipart uploads folder
avoid error bucket NotEmpty if multipart uploads folder exist
2022-06-29 16:21:16 +05:00
chrislu a8abab2412 Merge branch 'master' of https://github.com/chrislusf/seaweedfs 2022-06-29 02:44:14 -07:00
chrislu 449ecc8890 remove max connection age
related to https://github.com/chrislusf/seaweedfs/pull/3226
2022-06-29 02:44:12 -07:00
Chris Lu c6e6e303db
Merge pull request #3200 from lapshin-vitaly/bufix/validate-tags-on-copy
validate tags on copy object and add regex for validating tags
2022-06-29 01:37:14 -07:00
Lapshin Vitaliy 7c3d9d0535 remove println 2022-06-29 10:43:02 +03:00
Lapshin Vitaliy 61b2e3f9a2 fix return 2022-06-28 18:31:46 +03:00
Lapshin Vitaliy 606252472c fix return 2022-06-28 18:24:03 +03:00
Lapshin Vitaliy e969370913 fix parseTagging function and add test for validate tags 2022-06-28 14:48:55 +03:00